denshooter/genode
1
0
Fork 0
mirror of https://github.com/mmueller41/genode.git synced 2026-01-22 04:52:56 +01:00
Code Issues Packages Projects Releases Wiki Activity

Compare commits

base: denshooter:19.02
Branches Tags
denshooter:ealan denshooter:feature/habitats denshooter:ealan-v2 denshooter:gpgpu denshooter:pc-ixgbe denshooter:staging denshooter:genode-24 denshooter:master denshooter:tasking-profiler denshooter:gpgpu-bench
denshooter:22.05 denshooter:sculpt-22.04 denshooter:22.02 denshooter:21.11 denshooter:sculpt-21.10 denshooter:21.08 denshooter:21.05 denshooter:sculpt-21.03 denshooter:21.02 denshooter:20.11 denshooter:sculpt-20.08 denshooter:20.08 denshooter:20.05 denshooter:sculpt-20.02 denshooter:20.02 denshooter:19.11 denshooter:19.08 denshooter:sculpt-19.07 denshooter:19.05 denshooter:sculpt_ce denshooter:19.02 denshooter:18.11 denshooter:sculpt_vc denshooter:18.08 denshooter:18.05 denshooter:18.02 denshooter:17.11 denshooter:17.08 denshooter:17.05 denshooter:17.02 denshooter:16.11 denshooter:16.08 denshooter:16.05 denshooter:16.02 denshooter:15.11 denshooter:15.08 denshooter:15.05 denshooter:15.02 denshooter:14.11 denshooter:14.08 denshooter:14.05 denshooter:14.02 denshooter:13.11 denshooter:13.08 denshooter:13.05 denshooter:13.02 denshooter:12.11 denshooter:12.08 denshooter:12.05 denshooter:12.02 denshooter:11.11
..
compare: denshooter:sculpt-21.10
Branches Tags
denshooter:feature/habitats denshooter:ealan-v2 denshooter:gpgpu denshooter:pc-ixgbe denshooter:staging denshooter:genode-24 denshooter:master denshooter:ealan denshooter:tasking-profiler denshooter:gpgpu-bench
denshooter:22.05 denshooter:sculpt-22.04 denshooter:22.02 denshooter:21.11 denshooter:sculpt-21.10 denshooter:21.08 denshooter:21.05 denshooter:sculpt-21.03 denshooter:21.02 denshooter:20.11 denshooter:sculpt-20.08 denshooter:20.08 denshooter:20.05 denshooter:sculpt-20.02 denshooter:20.02 denshooter:19.11 denshooter:19.08 denshooter:sculpt-19.07 denshooter:19.05 denshooter:sculpt_ce denshooter:19.02 denshooter:18.11 denshooter:sculpt_vc denshooter:18.08 denshooter:18.05 denshooter:18.02 denshooter:17.11 denshooter:17.08 denshooter:17.05 denshooter:17.02 denshooter:16.11 denshooter:16.08 denshooter:16.05 denshooter:16.02 denshooter:15.11 denshooter:15.08 denshooter:15.05 denshooter:15.02 denshooter:14.11 denshooter:14.08 denshooter:14.05 denshooter:14.02 denshooter:13.11 denshooter:13.08 denshooter:13.05 denshooter:13.02 denshooter:12.11 denshooter:12.08 denshooter:12.05 denshooter:12.02 denshooter:11.11

2549 Commits
19.02 ... sculpt-21.

Author SHA1 Message Date
Christian Helmuth
abc9a2f232 depot: update recipe hashes 2021-10-14 13:46:25 +02:00
Norman Feske
0a35e02961 sculpt: documentation update 
Issue #4281
 2021-10-14 13:22:18 +02:00
Christian Helmuth
98ffa60577 Increase cap quota for system-shell terminal 2021-10-14 11:48:23 +02:00
Josef Söntgen
9980718c92 libyuv: build unoptimized on ARMv8 2021-10-14 11:46:42 +02:00
Josef Söntgen
2e0d9c8521 libdrm: short-cut tiling in iris back end 
Apparently the iris driver does not make use of tiling by the kernel, so
we shortcut the 'SET_TILING' call to keep iris happy with this quickfix.

However, tiling information may get lost, if the iris driver ever calls
'MMAP_GTT' and no fence is configured for the buffer. A follow-up commit
should address this shortcoming in the future.

Issue #4284
 2021-10-14 11:40:39 +02:00
Norman Feske
31567c8e46 sculpt: curate default index / installation 
- CPU-architecture annotations
- Change order of top-level menu, moving Mesa driver to the end
- Add black_hole, recall_fs, file_fault
- Add usb_webcam, test-capture
- Add audio driver and mixer
- Add vbox6, keeping vbox5-nova-sculpt as fallback
- Remove recall_fs launcher, which is obsolete with the recall_fs pkg
- Replace system_clock by system_clock-pc pkg

Issue #4281
 2021-10-14 11:02:18 +02:00
Norman Feske
006e807103 sculpt: version 21.10 2021-10-14 11:02:18 +02:00
Norman Feske
4dac2fd008 sculpt: increase RAM of leitzentrale nitpicker 
This follow-up commit to "sculpt: avoid flickering of leitzentrale"
allows nitpicker to double-buffer pixels during resize operation on a
screen size of 1920x1200.
 2021-10-14 11:02:17 +02:00
Norman Feske
68e8d1fd21 cached_fs_rom: remove diagnostic message 
The message "defer transfer..." spams the log on Sculpt OS.

Issue #4281
 2021-10-14 11:02:17 +02:00
Norman Feske
6a89f8b19e wm: fix key_cnt consistency glitch 
This patch presents all press and release events to the pointer state,
fixing the problem that _key_cnt was decreased but never increased.
However, the inconsistency had no observable effects in practice.

Issue #4176
 2021-10-14 11:02:17 +02:00
Norman Feske
ee463b21ae nitpicker: fix interplay of hover with dragging 
This patch extends the notion of having only one uniquely hovered client
in the presence of held keys.

If motion occurs once a key is pressed (e.g., while dragging), the
receiver of the key sequence observes the motion events. In this case,
we have to submit an artificial leave event to the originally hovered
client so that no more than one client observes itself as being hovered
at the same time. Once the key sequence is finished, the hovering is
updated again, eventually presenting a motion event to the originally
hovered client and a leave event to the receiver of the key sequence.

Issue #4176
 2021-10-14 11:02:17 +02:00
Norman Feske
1088035f8e wm: defer pointer report update 
This patch ensures that the pointer report is updated not before all
input events are handled. The change does not solve any observed
practical issue but the potential problem was spotted while reviewing
the code.

Issue #4176
 2021-10-14 11:02:17 +02:00
Christian Helmuth
b2ff2a2950 vbox6: configure OSS plugin only once (as /dev/dsp) 2021-10-14 11:02:16 +02:00
Stefan Kalkowski
c976a1d7e0 dde_linux: iterate over device names not types 
To clearly identify the correct device use the unique device's name
not the type. Otherwise a driver cannot drive several devices of the
same type.

Fix #4297
 2021-10-14 11:02:15 +02:00
Stefan Kalkowski
98400a68c9 os: extend ARM Platform::Device constructor 
Enable construction by explicitely naming a specific device,
if more than one device of the same type exist.

Ref #4297
 2021-10-14 11:02:15 +02:00
Josef Söntgen
8679f32d0b dde_linux: backport update event ring for usb_host 
This commit contains a backport of commit [1] that deals with updating
the event ring dequeue pointer more often to prevent unnecessary
'Event Ring Full' errors.

  [1] 'usb: host: xhci: update event ring dequeue pointer on purpose'
      (dc0ffbea5729a3abafa577ebfce87f18b79e294b)

Fixes #4296.
 2021-10-14 11:02:15 +02:00
Stefan Kalkowski
1d1379430a genode_c_api: safeguard session-policy parsing 
Fix #4294
 2021-10-14 11:02:15 +02:00
Stefan Kalkowski
b7a379546e genode_c_api: delay USB service announcement 
Ref #4294
 2021-10-14 11:02:14 +02:00
Stefan Kalkowski
873eb687b0 genode_c_api: reduce USB session ram requirements 
Ref #4294
 2021-10-14 11:02:14 +02:00
Stefan Kalkowski
1508fdc276 usb_block_drv: handle USB state at startup 
Fix #4293
 2021-10-14 11:02:14 +02:00
Sebastian Sumpf
c79cdc7b39 gpu/intel: use managed dataspace for aperture mappings 
Use 'Region_map_client' for aperture mappings through the GGTT instead
of 'Io_mem_connections'.

issue #4284
 2021-10-14 11:02:14 +02:00
Norman Feske
440debfc39 depot_query: filter index by CPU architecture 
This patch changes the depot_query tool to filter the returned index
data depending on the 'arch' as specified for the query. This way, one
index file can support multiple CPU architectures while allowing
individual entries to be architecture-specific.

Fixes #4295
 2021-10-14 11:02:14 +02:00
Norman Feske
28a71f4a73 cpu_balancer.run: adjust caps for sel4/x86_64 2021-10-14 11:02:14 +02:00
Christian Helmuth
4fd2c9c618 libuvc: exclude .git directory from src archive 2021-10-14 11:02:13 +02:00
Norman Feske
96997ead62 sculpt: fix warning during prepare step 
This patch adds the missing definition of 'prio_levels' in the prepare
sub init, fixing the warning:

  [init -> runtime -> prepare] Warning: vfs: invalid priority, upgrading from -2 to 0

Issue #4281
 2021-10-14 11:02:13 +02:00
Norman Feske
a12168e1bb pkg/mesa_gears: display frames per second 2021-10-14 11:02:13 +02:00
Sebastian Sumpf
d1461f6a72 vbox6: machine power-down support 
Use 'StateChange' event to check for machine's 'PowerOff' state, close
Gui connections and submit exit signal to EP which in turns calls exit.

Fixes #4291
 2021-10-14 11:02:13 +02:00
Christian Helmuth
c7abc9f983 vbox6: add pkg with webcam capture support 
Issue #4281
Issue #4287
 2021-10-14 11:02:13 +02:00
Christian Helmuth
d078f7db76 qemu-usb: adapt to webcam capture on/off changes 
Issue #4287
 2021-10-14 11:02:13 +02:00
Norman Feske
6f1d3862cd base: introduce Env::try_session 
The new 'Env::try_session' method mirrors the existing 'Env::session'
without implicitly handling exceptions of the types 'Out_of_ram',
'Out_of_caps', 'Insufficient_ram_quota', and 'Insufficient_cap_quota'.
It enables runtime environments like init to reflect those exceptions to
their children instead of paying the costs of implicit session-quota
upgrades out of the own pocket.

By changing the 'Parent_service' to use 'try_session', this patch fixes
a resource-exhaustion problem of init in Sculpt OS that occurred when
the GPU multiplexer created a large batch of IO_MEM sessions, with each
session requiring a second attempt with the session quota upgraded by
4 KiB.

Issue #3767
 2021-10-14 11:02:12 +02:00
Christian Prochaska
d5d7915b4d audio_in: fix compile errors when strict warnings are enabled 
Fixes #4290
 2021-10-14 11:02:11 +02:00
Christian Prochaska
b337ee2f2b vfs_oss: fix output fragment limit calculations 
Fixes #4289
 2021-10-14 11:02:11 +02:00
Johannes Schlatow
ef8a43c546 base-hw: cache maintenance 
touch each page before flushing to ensure that it's present in the page table

genodelabs/genode#4279
 2021-10-14 11:02:10 +02:00
Sebastian Sumpf
350353885e vbox6: add shared folder support 
builds virtualbox6-sharedfolders.lib.so from unmodified VirtualBox6
sources.

fixes #4288
 2021-10-14 11:02:10 +02:00
Norman Feske
eb80d6ce66 usb_webcam: on/off dependent on capture client 
This commit removes the report service from the usb_webcam pkg, which
was used to enable or disable the webcam driver. The on/off state is
instead controlled by the presence of a capture client. That is, once a
capture client appears, the webcam driver is started. Vice versa, once
no capture client exists, the webcam driver is removed automatically.

Internally, the detection of presence of a capture client is based on
nitpicker's 'displays' report, which is consumed as input of the
rom_filter, which in turn generates the configuration of a dynamic sub
init.

Fixes #4287
 2021-10-14 11:02:10 +02:00
Norman Feske
bc5a7eb495 rom_filter: support default input values 
Fixes #4286
 2021-10-14 11:02:10 +02:00
Stefan Kalkowski
cb67d07e61 lx_emul: be compliant to original softirq impl. 
Ref #4268
 2021-10-14 11:02:10 +02:00
Christian Helmuth
96ebed6c31 vbox6: use configuration from machine.vbox6 
There are some subtle incompatibilities in VirtualBox 6 with settings we
used in version 5. Therefore, the vbox6 package uses machine.vbox6 as
configuration file. An example configuration is provided by the
raw/vbox6 package.

Issue #4281
 2021-10-14 11:02:10 +02:00
Josef Söntgen
6093f8ad81 gpu/intel: deal with insufficient amount of CAPS 
'Out_of_ram' was so far the only exception a client had to deal with
during buffer managment. Allocating memory, however, does not only
consume RAM quota but CAP quota as well.

This commit tries to mitigate that shortcoming by reflecting the
'Out_of_caps' state back to the client. Furthermore it allows for
resource accounting on certain client allocations, e.g. buffers.

Fixes #4284.
 2021-10-14 11:02:09 +02:00
Josef Söntgen
f6d845e630 gpu: reflect CAP shortage during buffer management 
Allocating and mapping buffers not only consumes RAM quota, it consumes
CAP quota as well. Extended the Gpu session to allow for dealing with
that on the client side.

On a side note, the amount of initial CAP quota needed to establish
a connection is increased to cover the current costs of the Intel
GPU multiplexer.

Issue #4284.
 2021-10-14 11:02:09 +02:00
Josef Söntgen
27e55dab3e platform_drv: check quota before allocating buffer 
The platform driver uses a 'Constrained_ram_allocator' to allocate
meta-data on behave of a client. It uses the PD session as
'Ram_allocator' back end that in return is implemented via the
'Expanding_pd_session_client'.

Whenever the PD client itselft comes into resource shortage it will
ask its parent unconditionally. However, depending on the integration,
such a request may be left unanswered.

This commit introduces a check to prevent that situation from
occurring. In case the platform driver notices the resource shortage
it will reflect that back to the client.

Issue #4284.
 2021-10-14 11:02:09 +02:00
Norman Feske
05b451c563 pkg/mesa_gpu: use cached_fs_rom 
By using the cached_fs_rom instead of fs_rom, each client obtains the
same (read-only) copy of the shared library, avoiding a RAM resource
request when running mulitple GPU applications at the same time.
With this patch, Sculpt is able to host at least 3 Mesa applications.

Issue #4263
Issue #4281
 2021-10-14 11:02:09 +02:00
Sebastian Sumpf
2604c3cca6 vbox6: add mouse pointer shape support 
Implement 'MousePointerShapeChanged' event, retrieve parameters from
COM pointer, and update shape report.

fixes #4283
 2021-10-14 11:02:09 +02:00
Norman Feske
8ebc185caf depot: merge rtc_drv into system_clock runtime pkg 
This new version of the system_clock pkg does no longer depend on the
presence of an external 'Rtc' service as previously provided by the
Sculpt base system. Instead, it hosts the rtc_drv inside the subsystem.
Because rtc_drv is board-dependent, the system_clock pkg is named
system_clock-pc now.

Issue #4281
 2021-10-14 11:02:08 +02:00
Norman Feske
6e32102cc6 depot: add recall_fs runtime pkg 
Issue #4281
 2021-10-14 11:02:08 +02:00
Norman Feske
42541fcc92 sculpt: limit depot selection 
This patch reduces the items of the default depot-selection menu to
users that actively maintain a Sculpt index.

Issue #4281
 2021-10-14 11:02:08 +02:00
Norman Feske
7cae324726 sculpt: remove RTC driver from base system 
The RTC driver is not needed by the base system and can thereby be
handled as a user-installable component.

Issue #4281
 2021-10-13 14:50:46 +02:00
Sebastian Sumpf
1072a91592 qemu-usb: disable remote wake up in config descriptors 
Some guests don't handle remote wake up correctly causing devices to
stop functioning. Therefore, we disable the remote wake up bit (5) in
`bmAttributes` of the device configuration descriptor.

Thanks to Peter for the initial fix.

Fixes #4278
 2021-10-13 14:50:46 +02:00
Christian Helmuth
07bb3fc1ec Initial update of init-state ROM in init_loop test 
ROM clients have to request an initial update of dynamic ROMs explicitly
and should not depend on artificial signals from the ROM session on
signal-handler registration.

Issue #4274
 2021-10-13 14:50:46 +02:00
Johannes Schlatow
93583cce3b sequence: immediately stop child on exit 
The sequence app should immediately stop the child when it called
parent().exit(). Otherwise, the child will continue execution which
causes a race condition: The child's ld.lib.so will eventually destruct
an Attached_rom_dataspace for the config rom. If sequence destructed the
corresponding service first, we will get an Ipc_error.

genodelabs/genode#4267
 2021-10-13 14:50:46 +02:00
Josef Söntgen
e6bd9fd7da libc: implement SNDCTL_DSP_GETFMTS ioctl 
This I/O control is used to query the audio format, e.g. AFMT_S16_LE
in our case.

Fixes #4276.
 2021-10-13 14:50:46 +02:00
Martin Stein
870c5c7a81 file_vault: GUI control for encrypted virtual FS's 
Warning!

The current version of the file vault is not thought for productive use but
for mere demonstrational purpose! Please refrain from storing sensitive data
with it!

The File Vault component implements a graphical frontend for setting up and
controlling encrypted virtual file systems using the Consistent Block Encrypter
(CBE) for encryption and snapshot management. For more details see
'repos/gems/src/app/file_vault/README'.

Fixes #4032
 2021-10-13 14:50:46 +02:00
Martin Stein
3e375e4315 cbe tools: log less by default 
Previously unconditional calls to Genode::log in cbe init and the cbe trust
anchor VFS plugin were made dependent on a verbosity flag that is set to
"false" by default.

Ref #4032
 2021-10-13 14:50:46 +02:00
Martin Stein
8dfa586462 vfs/cbe_trust_anchor: fix bug in key decrypt 
The plugin used the ciphertext instead of the plaintext buffer for key
decryption which led to bogus ciphertext keys.

Ref #4032
 2021-10-13 14:50:46 +02:00
Martin Stein
a661aa79de vfs/cbe_trust_anchor: AES-key-wrap private key 
Instead of simply encrypting the private key with AES-256 when storing it to
the 'encrypted_private_key' file, wrap it using the AES-key-wrap algorithm
described in RFC 3394 "Advanced Encryption Standard (AES) Key Wrap Algorithm".
This is more secure and enables us to directly check whether the passphrase
entered by the user was correct or not.

Ref #4032
 2021-10-13 14:50:46 +02:00
Martin Stein
94701eec09 vfs/cbe_trust_anchor: rename secured_superblock 
As the file formerly named 'secured_superblock' actually contains the hash of
the superblock that was secured, it was renamed 'superblock_hash'.

Ref #4032
 2021-10-13 14:50:46 +02:00
Martin Stein
a28ca44101 vfs/cbe_trust_anchor: rename keyfile 
As the file formerly named 'keyfile' actually contains the encrypted private
key of the Trust Anchor, it was renamed 'encrypted_private_key'.

Ref #4032
 2021-10-13 14:50:46 +02:00
Martin Stein
641a738e3d vfs/cbe_trust_anchor: encrypt symmetric keys 
By now, the symmetric keys were only XOR'ed with the private key as placeholder
for a real encryption. Now they are encrypted using AES256 with the TA's
private key as key

Ref #4032.
 2021-10-13 14:50:46 +02:00
Martin Stein
f41dec67e1 vfs/cbe_trust_anchor: use secure private key 
A private key of 256 bits is generated pseudo-randomly using the jitterentropy
VFS plugin on initialization. The private key is stored in the key file
encrypted via AES256 using the SHA256 hash of the users passphrase. When
unlocking the CBE device, the encrypted private key is read from the key file
and decrypted with the hash of the users passphrase.

Ref #4032
 2021-10-13 14:50:46 +02:00
Martin Stein
75a55b62a3 vfs/cbe_trust_anchor: use hash of passphrase 
Instead of using the user passphrase directly, use its SHA256 hash calculated
using libcrypto. The passphrase hash is still stored in the key file to be
used as base for the very primitive way of generating the private key.

Ref #4032
 2021-10-13 14:50:46 +02:00
Martin Stein
989b7f39e1 vfs/cbe_trust_anchor: use secure symmetric keys 
Use the jitterentropy vfs plugin for the generation of new symmetric block
encryption keys in the cbe trust anchor vfs plugin.

Ref #4032
 2021-10-13 14:50:46 +02:00
Martin Stein
a2d2b874ec vfs/cbe_trust_anchor: sync keyfile-handle close 
Closing the keyfile handle after a write operation wasn't synchronised to the
actual end of the write operation.

Issuing a write operation at the back end returns successfull as soon as the
back end has acknowledged that it will execute the operation. However, the
actual writing of the data might still be in progress at this point. But the
plugin used to close the file handle and declare the operation finished at this
point which led to warnings about acks on unknown file handles and leaking
resources. Now, the plugin issues a sync operation directly after the write
operation and waits for the sync to complete. This ensures that the plugin
doesn't declare the operation finished too early.

Ref #4032
 2021-10-13 14:50:45 +02:00
Martin Stein
42490208c2 vfs/cbe_trust_anchor: fix unlocking 
The unlocking operation in the trust anchor was broken wich caused bad keys in
the CBE. This rewrites the whole operation to work as desired. Note that this
doesn't make it more safe! The private key is still almost the same as the
passphrase and stored plaintext.

Ref #4032
 2021-10-13 14:50:45 +02:00
Martin Stein
95639a7492 vfs/cbe_trust_anchor: close handles correctly 
The plugin used to close file handles via the 'vfs_env.root_dir.close'.
However, this lead to resource leaks and apparently isn't the right way to
do it. Other VFS plugins do it by calling 'close' directly on the handle and
doing it in the trust anchor plugin also, fixes the leaks.

Ref #4032
 2021-10-13 14:50:45 +02:00
Martin Stein
df7de17435 vfs/cbe_trust_anchor: sync hashfile-handle close 
Closing the hashfile handle after a write operation wasn't synchronised to the
actual end of the write operation.

Issuing a write operation at the back end returns successfull as soon as the
back end has acknowledged that it will execute the operation. However, the
actual writing of the data might still be in progress at this point. But the
plugin used to close the file handle and declare the operation finished at this
point which led to warnings about acks on unknown file handles and leaking
resources. Now, the plugin issues a sync operation directly after the write
operation and waits for the sync to complete. This ensures that the plugin
doesn't declare the operation finished too early.

Ref #4032
 2021-10-13 14:50:45 +02:00
Martin Stein
1b4a80ffae vfs/cbe: control/deinitialize file 
There were no means for issuing a Deinitialize request at the CBE using the
CBE VFS plugin. The new control/deinitialize file fixes this. When writing
"true" to the file, a Deinitialize request is submitted at the CBE. When
reading the file, the state of the operation is returned as a string of the
format "[current_state] last-result: [last_result]" where [current_state] can
be "idle" or "in-progress" and [last_result] can be "none", "success", or
"failed".

Ref #4032
 2021-10-13 14:50:45 +02:00
Martin Stein
88e6991083 vfs/cbe: fix ID argument on discard_snap 
When discarding a snapshot, the CBE VFS plugin didn't communicate the ID of
the snapshot to the CBE. Instead it set the ID argument to 0. Therefore the
operation never had any effect.

Ref #4032
 2021-10-13 14:50:45 +02:00
Martin Stein
9b953e79ba vfs/cbe: support watching the snapshots fs 
The snapshots file system couldn't be watched. But it's of interest to watch
for appearing and disappearing snapshots.

Ref #4032
 2021-10-13 14:50:45 +02:00
Martin Stein
4c1eae97cf vfs/cbe: fix result of SnapshotsFS.num_dirent("/") 
The snapshots file system used to return the number of snapshots on
'num_dirent' when called for the root directory although it was expected to
return 1. This confused the tooling ontop of the VFS.

Ref #4032
 2021-10-13 14:50:45 +02:00
Martin Stein
f051bfa90d vfs/cbe: support watching "rekey" file 
The control/rekey file couldn't be watched although it was meant to be used
to watch the current state of the rekey operation.

Ref #4032
 2021-10-13 14:50:45 +02:00
Martin Stein
4ed45dd303 vfs/cbe: support watching "extend" file 
The control/extend file couldn't be watched although it was meant to be used
to watch the current state of the extend operation.

Ref #4032
 2021-10-13 14:50:45 +02:00
Martin Stein
2d0ac161cb vfs/cbe: mark extend/rekey fs readable 
Despite being readable, the files control/extend and control/rekey proclaimed
that they were not when asked. This caused the fs_query tool to not report the
content of the files although it could have.

Ref #4032
 2021-10-13 14:50:45 +02:00
Martin Stein
34245376ba vfs/cbe: fix size of extend/rekey fs 
Stat calls on the control/extend and control/rekey files returned a bogus file
size that led to an error in the VFS File_content tool. The tool complained
that the size of the file determined while reading the content differs from the
one reported by the stat operation. Now, the stat call will always determine
the actual size of what would be read.  However, it isn't guaranteed that this
size doesn't change in the time after the stat operation and before the read
operation.

Ref #4032
 2021-10-13 14:50:45 +02:00
Martin Stein
bd79f93657 cbe/types: invalid generation, generation string 
Ref #4032
 2021-10-13 14:50:45 +02:00
Martin Stein
731df4b037 recipes/src/cbe: build also cbe_init_trust_anchor 
Ref #4032
 2021-10-13 14:50:45 +02:00
Martin Stein
d3cc5c9cd2 gems/recipes: api/cbe 
Ref #4032
 2021-10-13 14:50:45 +02:00
Martin Stein
ff656401b3 gems: update cbe port 
This update fixes a bug when discarding snapshots.

Ref #4032
 2021-10-13 14:50:45 +02:00
Christian Helmuth
46ee872b50 vbox6: shared clipboard service 
The service is loaded dynamically VBoxSharedClipboard.so at runtime. The
VFS configuration mounts the shared object at /VBoxSharedClipboard.so as
the file is checked by contrib code before loading. An init
configuration in pkg/vbox6/runtime illustrates this and how to re-label
the VBoxSharedClipboard.so ROM to its real name
virtualbox6-sharedclipboard.lib.so.
 2021-10-13 14:50:45 +02:00
Christian Helmuth
5aee693f70 vbox6: limit wait for ack to 15 ms in AHCI model 
During Windows 10 boot with sequential block requests, the AHCI request
worker finished earlier than the EMT thread signals hEvtProcess and
begins waiting for hEvtProcessAck indefinitely. The timeouts helps to
survive this short phase.

A better solution would use conditional variables, which are not
provided in VirtualBox's runtime.
 2021-10-13 14:50:45 +02:00
Josef Söntgen
f21cf3f8b1 mesa: name driver library appropriately 
Rename the 'egl_drv' library and the various back ends to 'mesa_gpu_drv'
and 'mesa_gpu-<backend>'.

Fixes #4275.
 2021-10-13 14:50:37 +02:00
Norman Feske
32c283d26f os: introduce C API for generating Genode events 
This patch introduces a C API to be used by input drivers to generate
Genode events. The initial version is limited to multitouch events only.

Fixes #4273
 2021-10-13 14:46:54 +02:00
Stefan Kalkowski
f4cb5cc299 dde_linux: preserve Io_signal_handler for IRQs 
Do not construct/destruct signal handlers for interrupts dynamically,
but only the Irq session to prevent a deadlock.

Fix #4272
 2021-10-13 14:46:54 +02:00
Stefan Kalkowski
6ae55d490b lx_emul: restrict allocations to minimal alignment 
* Use the architecture-dependent minimal alignment for all allocations,
  e.g. on ARM it is necessary to have cacheline aligned allocations for DMA
* Remove the allocation functions without alignment from generic API
* Fix a warning

Fix #4268
 2021-10-13 14:46:54 +02:00
Stefan Kalkowski
2ac8620f44 lx_emul: invalidate cache for DMA-read only 
After a DMA transaction do only invalidate cachelines from the
corresponding DMA buffers if data got transfered from device to
CPU, and not vice versa. Otherwise it might result in data corruption.

Ref #4268
 2021-10-13 14:46:54 +02:00
Stefan Kalkowski
d8c344f6b9 lx_emul: do not block kworkers unconditionally 
Ref #4268
 2021-10-13 14:46:54 +02:00
Stefan Kalkowski
f8cf0442ed lx_emul: implement lookup of dma-addr to virt-addr 
* Use the new Lx_kit::Map as lookup structure for virt-to-dma and vice versa,
  instead of a list-based registry

Ref #4268
 2021-10-13 14:46:54 +02:00
Stefan Kalkowski
9bc7ecb605 lx_emul: provide alignment for DMA allocations 
Ref #4268
 2021-10-13 14:46:54 +02:00
Christian Helmuth
4c4ce2f899 report_rom: versioning and explicit notification 
The former implementation did not internally track ROM changes notified
vs. delivered to the client. We adapt the versioning implementation
implemented in dynamic_rom_session.h and enable explicit notification of
the current version.

The feature is used by the clipboard to notify permitted readers of the
clipboard ROM service on focus change via the newly created private
Rom::Module::_notify_permitted_readers() function.

Fixes #4274
 2021-10-13 14:46:54 +02:00
Stefan Kalkowski
4803937dd2 os: introduce C-API to provide USB service 
Fix #4270
 2021-10-13 14:46:54 +02:00
Stefan Kalkowski
6789b86871 base-hw: optimize cache maintainance for ARMv8 
* Remove the data-synchronization barrier from the inner-loop
* Instead add a system-wide barrier at the end of the operation

Fix #4269
 2021-10-13 14:46:54 +02:00
Martin Stein
9542bcf88c base-hw: reduce includes in board.h 
The includes for the address-space-ID allocator and the translation table are
usually specific to the CPU in use. Therefore these includes can be moved from
their current location in the board header to the CPU headers. This reduces the
number of decisions a board maintainer has to make if the CPU model he's aiming
for is already available.

This can probably also be applied for other includes in the board headers but I
intentionally leave it for a future commit as I don't have the time to do it
all now.

Ref #4217
 2021-10-13 14:46:54 +02:00
Martin Stein
ad059362d2 base-hw: reduce hardware-specific include paths 
For base-hw Core, we used to add quite some hardware-specific include paths
to 'INC_DIR'. Generic code used to include, for instance, '<cpu.h>' and
'<translation_table.h>' using these implicit path resolutions. This commit
removes hardware-specific include paths except for

1) the '<board.h>' include paths (e.g., 'src/core/board/pbxa9'),
2) most architecture-specific include paths (e.g., 'src/core/spec/arm_v7'),
3) include paths that reflect usage of virtualization or ARM Trustzone
   (e.g., 'src/core/spec/arm/virtualization').

The first category is kept because, in contrast to the former "spec"-mechanism,
the board variable used for this type of resolution is not deprecated and the
board headers are meant to be the front end of hardware-specific headers
towards generic code which is why they must be available generically via
'<board.h>'.

The second category is kept because it was suggested by other maintainers that
simple arch-dependent headers (like for the declaration of a CPU state) should
not imply the inclusion of the whole '<board.h>' and because the architecture
is given also without the former "spec"-mechanism through the type of the build
directory. I think this is questionable but am fine with it.

The third category is kept because the whole way of saying whether
virtualization resp. ARM Trustzone is used is done in an out-dated manner and
changing it now would blow up this commit a lot and exceed the time that I'm
willing to spend. This category should be subject to a future issue.

Ref #4217
 2021-10-13 14:46:53 +02:00
Martin Stein
49b09702b8 base-hw: don't include board.h from hardw. headers 
The 'src/core/board/<board>/board.h' header is thought as front end of
hardware-specific headers of a given board towards the generic base-hw Core
code. Therefore it leads to problems (circular includes) if the board.h header
is included from within another hardware-specific header.

If hardware-specific headers access declarations from namespace Board in a
definition, the definition should be moved to a compilation unit that may
include board.h. If hardware-specific headers access declarations from board.h
in a declaration, they should either use the primary declaration from the
original header or, if the declaration must be selected according to the board,
another board-specific header should be introduced to reflect this abstraction.

This is applied by this commit for the current state of base-hw.

Ref #4217
 2021-10-13 14:46:53 +02:00
Martin Stein
5d74509b2d base-hw: get rid of static perf counter object 
It is not necessary to have a class, an object, and a generic header for the
perfomance counter. The kernel merely enables the counter using cpu registers
('msr' instructions, no MMIO) on arm_v6 and arm_v7 only. Therefore this commit
makes the header arm-specific and replaces class and global static object with
a function for enabling the counter.

Fixes #4217
 2021-10-13 14:46:53 +02:00
Martin Stein
b817e1977c base-hw: serial output as Main member 
Let the kernel's serial-output driver be a member of the one Kernel::Main
object instead of having it as global static variable.

Ref #4217
 2021-10-13 14:46:53 +02:00
Martin Stein
910788313e base-hw: get rid of static variable in Kernel::Pd 
Ref #4217
 2021-10-13 14:46:53 +02:00
Martin Stein
02f00a999c base-hw: global IRQ controller as Main member 
Let the kernel's driver for the global IRQ controller be a member of the one
Kernel::Main object instead of having it as static variables in the drivers for
the local IRQ controllers. Note that this commit spares out renaming 'Pic' to
'Local_interrupt_controller' which would be more sensible now with the new
'Global_interrupt_controller' class. Furthermore, on ARM boards the commit
doesn't move 'Distributer' stuff to the new global IRQ controller class as they
don't have real data members (only MMIO) and can be instanciated for each CPU
anew. However, the right way would be to instanciate them only once in Main as
well.

Ref #4217
 2021-10-13 14:46:53 +02:00
Martin Stein
441d137482 base-hw: Address-Space-ID allocator as Main member 
Let the kernel's Address-Space-ID allocator be a member of the one
Kernel::Main object instead of having it as global static variable.

Ref #4217
 2021-10-13 14:46:53 +02:00
Martin Stein
0c61b25bcf base-hw: no unmanaged_singleton in kernel thread 
The unmanaged-singleton approach was used in this context only because of the
alignment requirement of the Core main-UTCB. This, however can also be achieved
with the new 'Aligned' utility, allowing the UTCB to be a member of the Core
main-thread object.

Ref #4217
 2021-10-13 14:46:53 +02:00
Martin Stein
b6641eaa25 base-hw: Core PD as Main member 
Let the Core protection-domain object be a member of the one Kernel::Main
object instead of having it as global static variable.

Ref #4217
 2021-10-13 14:46:53 +02:00
Martin Stein
d6e347163d base-hw: unnecessary unmanaged-singleton includes 
Remove some deprecated include directives for the unmanaged-singleton header.

Ref #4217
 2021-10-13 14:46:53 +02:00
Martin Stein
5dd8ee5840 base-hw: global IRQ pool as Main member 
Let the global kernel IRQ-pool be a member of the one Kernel::Main object
instead of having it as global static variable.

Ref #4217
 2021-10-13 14:46:53 +02:00
Martin Stein
c49db16762 base-hw: don't use boot_info() outside main.cc 
It's sufficient to access the boot info only on kernel initialization time.
Therfore, it can remain completely hidden to the rest of the kernel inside
kernel/main.cc in the initialization function.

Ref #4217
 2021-10-13 14:46:53 +02:00
Martin Stein
b247def09a base-hw: Core main-thread as Main member 
Let the Core main-thread object be a member of the one Kernel::Main object
instead of having it as global static variable.

Ref #4217
 2021-10-13 14:46:53 +02:00
Martin Stein
2b89cd66cb base-hw: kernel CPU-pool as Main member 
Let the kernel CPU-pool be a member of the one Kernel::Main object instead of
having it as global static variable.

Ref #4217
 2021-10-13 14:46:53 +02:00
Martin Stein
be3d5232c8 base-hw: kernel-data lock as Main member 
Let the global kernel-data lock be a member of the one Kernel::Main object
instead of having it as global static variable.

Ref #4217
 2021-10-13 14:46:53 +02:00
Martin Stein
82cf31ac27 base-hw: introduce Kernel::Main class 
This commit introduces the Kernel::Main class that replaces the former way of
initializing the kernel (former 'kernel_init' function) and calling the C++
kernel entry handler (former 'kernel' function). These two are now
'Main::initialize_and_handle_kernel_entry' and 'Main::handle_kernel_entry'.
Also reading the execution time of the idle threads was already moved to
'Main'. The one static Main instance is meant to successivly replace all the
global static objects of the base-hw kernel with data members of the Main
instance making the data model of the kernel much more comprehensible. The
instance and most of its interface are hidden in kernel/main.cc. There are only
rare cases where parts of the Main interface must be accessible from the
outside. This should be done in the most specific way possible (see main.h)
and, if possible, without handing out references to Main data members or the
Main instance itself.

Ref #4217
 2021-10-13 14:46:52 +02:00
Martin Stein
9de43a48b6 base-hw: conform pc board header 
Normally, the board header can be found for each supported board under
'src/core/board/<BOARD>/board.h'. This was not the case for the board 'pc'
that was located under 'src/core/spec/x86_64/board.h'. The commit fixes this.

Ref #4217
 2021-10-13 14:46:52 +02:00
Martin Stein
a9563bfd4c base-hw: rename Core_thread Core_main_thread 
The class name Core_thread in Kernel for the object of the first thread of
core is too generic as there can be an arbitrary number of threads in core
besides this one. Furthermore, creating a core thread has its own syscall
'new_core_thread' that isn't related in any way to Core_thread. Therefore
this commit introduces the more specific name Core_main_thread as replacement
for Core_thread.

Ref #4217
 2021-10-13 14:46:52 +02:00
Martin Stein
693a2b5421 base-hw: get rid of cpu_pool() in platform.cc 
The function was only still used for reading the execution time of idle threads
of CPUs. Certainly, it is technically fine and more performant to read these
values directly from the kernel objects without doing a syscall. However,
calling cpu_pool() for it provides read and write access to a lot more than
only the execution time values. The interface via which Core directly reads
state of the kernel should be as narrow and specific as possible.
Perspectively, we want to get rid of the cpu_pool() accessor anyway. Therefore
this commit introduces Kernel::read_idle_thread_execution_time(cpu_idx) as
replacement. The function is implemented in kernel code and called by Core in
platform.cc.

Ref #4217
 2021-10-13 14:46:52 +02:00
Martin Stein
277adcacb0 base-hw: make Trace_source in Platform() a class 
Apparently, there is no need for exposing the data members of Trace_source, so,
we sould better make them private before someone gets the impression that they
are meant to be accessed directly.

Ref #4217
 2021-10-13 14:46:52 +02:00
Martin Stein
aa6a7db50a base-hw: communicate kernel irqs via boot info 
Core used to read the kernel-reserved IRQs from the timer objects in the
kernel's CPU objects and the PIC class (inter-processor IRQ). Besides not
being "good style" to access a kernel object in Core, this becomes a problem
when trying to prevent CPU pool from being accessed via global functions.

As a solution, this commit extends the boot info to also carry an array of all
kernel-reserved IRQs.

Ref #4217
 2021-10-13 14:46:52 +02:00
Martin Stein
8b69bc96f9 base-hw: separate variants of Kernel_object(...) 
For the constructor of Kernel_object<T> there are two variants. One for the
case that it is called from Core where the kernel object (type T) must be
created via a syscall and one when it is called from within the kernel and the
kernel object can be created directly. Selecting one of these variants was done
using a bool argument to the constructor. However, this implies that the
constructor of Kernel_object<T> and that of T have the same signature in the
variadic arguments, even in the syscall case, although technically it would
then not be necessary.

This becomes a problem as soon as kernel objects created by Core shall receive
additional arguments from the kernel, for instance a reference to the global
CPU pool, and therefore stands in the way when wanting to get rid of global
statics in the kernel. Therefore, this commit introduces two constructors that
are selected through enum arguments:

! Kernel_object(Called_from_kernel, ...);
! Kernel_object(Called_from_core, ...);

Ref #4217
 2021-10-13 14:46:52 +02:00
Martin Stein
6e4ef43bf0 base-hw: always use 'unsigned' for priorities 
At some points in the code, 'signed' was used instead of the more appropriate
'unsigned' type.

Ref #4217
 2021-10-13 14:46:52 +02:00
Martin Stein
b922dc5c10 base-hw: fix self-include in kernel/types.h 
That this header apparently used to include itself didn't cause any problems
so far but still remained bad style.

Ref #4217
 2021-10-13 14:46:52 +02:00
Josef Söntgen
cacb83b163 gpu: introduce mapping attributes 
The various mapping methods are modelled after the requirements of
the Intel GPUs or rather the Mesa driver back end.

With upcoming support for other driver back ends, we need to
sequeeze their requirements in as well. For now hijack 'map_buffer'
to provide for specifying the kind of attributes the client needs.

For now all buffers mapped in the GGTT for Intel GPUs are treated
as RW.

Issue #4265.
 2021-10-13 14:46:52 +02:00
Josef Söntgen
90e151e2c4 gpu: add 'complete' RPC call 
This call allows for checking if the given execution buffer has been
completed and complements the completion signal. Initially the GPU
multiplexer always sent such a signal when the currently scheduled
execution buffer has been completed. During enablement of the 'iris'
driver it became necessary to properly check of sequence number.

In case of the Intel GPU multiplexer the sequence numbers are
continous, which prompted the greater-than-or-equal check in the
DRM back end. By hidding this implementation detail behind the
interface, GPU drivers are free to deal with sequence numbers any
way they like and allows for polling in the client, where the
completion signal is now more of a progress signal.

Issue #4265.
 2021-10-13 14:46:52 +02:00
Josef Söntgen
e37792ce94 gpu: introduce information dataspace 
The current info implementation (as RPC) is limited in a few ways:

  * The amount of data that may be transferred is constrained by the
    underlying base platform
  * Most information never changes during run time but is copied
    nonetheless
  * The information differs depending on the used GPU device and
    in its current implementation only contains Intel GPU specific
    details

With this commit the 'info' RPC call is replaced with the
'info_dataspace' call that transfers the capability for the dataspace
containing the information only. This is complemented by a client
local 'attached_info' call that allows for getting typed access to
the information. The layout of the information is moved to its own
and GPU-specific header file, e.g., 'gpu/info_intel.h'

Issue #4265.
 2021-10-13 14:46:52 +02:00
Josef Söntgen
cfb170c719 gpu: move exec sequence number to Gpu namespace 
Moving the type definition removes the aritifial linkage to the Gpu
information structure.

Issue #4265.
 2021-10-13 14:46:52 +02:00
Josef Söntgen
3b40790e02 gpu: use Buffer_id to identify buffers 
Rather than using the dataspace capability directly, let the client
choose its own local identifier that is linked to the underlying
capability.

Fixes #4265.
 2021-10-13 14:46:52 +02:00
Stefan Kalkowski
9a80c3a618 depot: suppress error when copying empty package 
Instead of failing, ignore empty packages when copying over
depot packages to a run-script environment.

Fix #4271
 2021-10-13 14:46:52 +02:00
Josef Söntgen
ba430dfeac libdrm/iris: destroy Buffer_handle object 
When the client frees the buffer, destroy the corresponding
Buffer_handle as well.

Fixes #4266.
 2021-10-13 14:46:52 +02:00
Piotr Tworek
b157256a2b uplink client: Add missing space in warning message. 
Right now the warning about failure to forward packet from driver to
uplink RX connection reads:

  "exception while trying to forward packet from driverto Uplink
   connection TX"

Add missing space between "driver" and "to".

Issue #4264
 2021-10-13 14:46:52 +02:00
Piotr Tworek
60980045ea virtio_nic: Increase default TX & RX virtqueue sizes. 
32KB is a rather small value. The driver can cope with it now, but
it does not perform as well as it should. This visible especially
in scenarions like nic_router_flood where we still often hit
synchronous wait path. Bump the size to 256kB.

Issue #4264
 2021-10-13 14:46:51 +02:00
Piotr Tworek
880cd3a490 virtio_nic: Fix packet transmission handling. 
The problem can be seen when running nic_router_flood scenarion on arm
qemu_virt boards. With the amount of data this scenario tries to send
the driver quickly complains it has failed to push data into TX VirtIO
queue. After this warning message is printed nothing really happens and
after a while the test scenario fails.

The fact that we can't write all available data to the device is not
unexpected. VirtIO queue size is slected at initialization time and we
don't change it during driver lifetime. It can be tweaked via driver
config, but this does not change the fact that we'll always be able to
produce more data packets than we have free space in the VirtIO queue.

IMO the expected behavior of the driver in such case should be to:
1. Notify the device there is data to process.
2. Wait for the device to process at least part of it.
3. Retry sending queued packets.

One could expect returning Transmit_result::RETRY from _drv_transmit_pkt
would produce such result. Unfortunately it seems that Uplink_client_base
treats RETRY return value as indication of link being down. It'll retry
sending the packet only after the device notifies it the link is once
again up. This is the reason why nothing happens when running
nic_router_flood on top of virtio_nic driver. The link never goes down
in this case so once we fill the TX VirtIO queue and tell the base class
to retry the send, we'll be stuck waiting for link up change event
which will never arrive.

To fix this problem, when sending a packet to the device fails, do a
synchrnonus TX VirtIO queue flush (tell device there is data to process
and wait until its done with it).

With this fix in place nic_router_flood test scenario passes on both arm
qemu_virt boards.

Issue #4264
 2021-10-13 14:46:51 +02:00
Piotr Tworek
b7f66626c2 virtio: Make avail and used ring pointers volatile 
The contents of those descriptor rings can be modified by the device.
Mark them as volatile so the compiler does not make any assumptions
about them.

Issue #4264
 2021-10-13 14:46:51 +02:00
Christian Helmuth
c0a7696c71 tool/dts/extract: convert regex strings to latin1 
The former encoding was UTF-8, which works quite well if LC_CTYPE is
ensured to be an UTF-8 codeset (e.g., en_US.UTF-8 or C.UTF-8 . But, if
LC_CTYPE is set to C or latin1 for example, the Tcl regex library enters
an infinite loop because of unexpected characters used as markers
n the strings (e.g., SECTION SIGN U+00A7).

Therefore, the extract tool was converted to latin1 with the following
commands and now works for LC_CTYPE C and UTF-8 codesets.

   iconv -f utf-8 -t latin1 tool/dts/extract > /tmp/e
   cp /tmp/e tool/dts/extract
 2021-10-13 14:46:51 +02:00
Alexander Boettcher
d85a448c52 gpu/intel: provide supported devices as config 
Adjust drivers_managed and sculpt accordingly.

Issue #4260
 2021-10-13 14:46:51 +02:00
Sebastian Sumpf
d67899be95 gpu/intel: enable 32 bit CPU support 
* use Gpu::addr_t (64 Bit) where necessary instead of Genode::addr_t.

issue #4260
 2021-10-13 14:46:51 +02:00
Alexander Boettcher
6112c0df6d mesa: rename gpu-* pkg to mesa_gpu-* 
for easier linking by user with the Mesa client ROM requests, which will be
mesa_gpu-drv.lib.so. Adjust mesa-gears demo accordingly.

Issue #4263
 2021-10-13 14:46:51 +02:00
Alexander Boettcher
367385aed7 gpu/intel: add more supported Intel GPUs 
- Lenovo T470p, T490, T490s

Issue #4260
 2021-10-13 14:46:51 +02:00
Alexander Boettcher
0f72356570 gpu/intel: use read out gttm size 
and don't assume 8M, which leads to Region_conflicts if size is >8M (X201).

Issue #4260
 2021-10-13 14:46:51 +02:00
Christian Helmuth
727fa86088 autopilot: support additional repo directories 
-a <repo-dir> appends repo-dir to REPOSITORIES in build.conf
 2021-10-13 14:46:51 +02:00
Christian Helmuth
32394c0733 hello: fix resource config / add to autopilot list 2021-10-13 14:46:51 +02:00
Alexander Boettcher
5aa3c56e5c mesa: add gears package 
Issue #4263
 2021-10-13 14:46:51 +02:00
Alexander Boettcher
0c2edce8ac sculpt: add Gpu service support 
Fixes #4263
 2021-10-13 14:46:51 +02:00
Alexander Boettcher
2d2ef2a763 mesa: add gpu-iris package providing IRIS driver 
Issue #4263
 2021-10-13 14:46:51 +02:00
Alexander Boettcher
b6beff673a mesa: add gpu-cpu package providing SWRAST driver 
Issue #4263
 2021-10-13 14:46:51 +02:00
Sebastian Sumpf
07881f90a9 mesa/libdrm: remove signal EP 
This is a left over from Mesa-11 and we exchanged it with a
'wait_and_dispatch_one_io_signal' for synchronous signal waits.

issue #4260
 2021-10-13 14:46:50 +02:00
Sebastian Sumpf
3daa94ff2e gpu/intel: Add support for Gen9+ 
This commit contains features and buf fixes:

* Catch errors during resource allocation

* Because Mesa tries to allocate fence (hardware) registers for each
  batch buffer execution, do not allocate new fences for buffer objects
  that are already fenced

* Add support for global hardware status page. Each context additionally
  has a per-process hardware status page, which we used to set the
  global hardware status page during Vgpu switch. This was obviously
  wrong. There is only one global hardware status page (set once during
  initialization) and a distinct per-process page for contexts.

* Write the sequence number of the currently executing batch buffer to
  dword 52 of the per-process hardware status page. We use the pipe line
  command with QW_WRITE (quad word write), GLOBAL_GTT_IVB disabled
  (address space is per-process address space), and STORE_DATA_INDEX
  enabled (write goes to offset of hardware status page). This command
  used to write to the scratch page. But Linux now uses the first
  reserved word of the per-process hardware status page.

* Add Gen9+ WaEnableGapsTsvCreditFix workaround. This sets the "GAPS TSV
  Credit fix Enable" bit of the Arbiter control register (GARBCNTLREG)
  as described by the documentation this bit should be set by the BIOS
  but is not on most Gen9/9.5 platforms. Not setting this bit leads to
  random GPU hangs.

* Increase the context size from 20 to 22 pages for Gen9. On Gen8 the
  hardware context is 20 pages (1 hardware status page + 19 ring context
  register pages). On Gen9 the size of the ring context registers has
  increased by two pages to 21 pages or 81.3125 KBytes as the IGD
  documentation states.

* The logical ring size in the ring buffer control of the execlist
  context has to be programmed with number of pages - 1. So 0 is 1 page.
  We programmed the actual number of pages before, leading to ring
  buffer execution of NOOPs if page behind our ring buffer was empty or
  GPU hangs if there was data on the page.

issue #4260
 2021-10-13 14:46:50 +02:00
Sebastian Sumpf
59b23bc7e1 libdrm: Iris - execute batch buffer synchronous 
* Wait for for completion before return from 'execbuffer2'. This makes
  buffer execution synchronous.

* Because the Iris driver manages the virtual address space of the GPU
  and creates one GEM context for each batch buffer we have to map/unmap
  all buffer objects before and after batch buffer execution.

issue #4260
 2021-10-13 14:46:50 +02:00
Norman Feske
b7bb6869b4 drivers_interactive-pc: remove platform service 
The platform driver should better stay internal to the drivers subsystem
to reinforce the consistency with other drivers_interactive packages.
 2021-10-13 14:45:24 +02:00
Norman Feske
e6cdaafb20 tool/ports: use current time as modification time 
Fixes #4262
 2021-10-13 14:02:16 +02:00
Norman Feske
d072c408ab dde_linux: call init_page_count for each page 
The lx_emul_virt_to_pages implementation initialized the page ref
counter only for the first page, leaving the remaining elements in
uninitialized state. This, in turn, rendered the Linux page_pool (as
used by the emac network driver) ineffective, ultimately leading the a
memory leak. The fix changes the call of 'init_page_count' to take the
loop variable as argument.

Issue #4225
 2021-10-13 14:02:16 +02:00
Johannes Schlatow
4dacac3dbb depot_autopilot: fix test-trace 
Increased number of trace subjects since the test sporadically fails on
some platforms.

Also added a sanity check to print an error message in case we run into
the same issue again.

Fixes genodelabs/genode#4261
 2021-10-13 14:02:16 +02:00
Norman Feske
914a41a8bd dde_linux: adjustments for Linux 5.14.1 
Issue #4259
 2021-10-13 14:01:03 +02:00
Tomasz Gajewski
e6c915ae06 libssh port: immediate mode for ssh poll 
This patch adds a switch to internal poll function in libssh that
allows to force this function to immediately return without actually
polling for data and in consequence processing this data. This switch
is used to avoid calling callback functions when flushing output
streams which caused locks due to recursive access to internal
ssh_terminal sessions registry.

Issue #4258
 2021-10-13 14:01:02 +02:00
Tomasz Gajewski
6ef6f16cb3 libssh port: backported sftp_server_free 
sftp_server_free function was added in 0.9 version of libssh and is
required to avoid memory leaks when clients are disconnecting.

Issue #4258
 2021-10-13 14:01:02 +02:00
Tomasz Gajewski
f327a40bbb libssh port: required modifications for sftp 
This patch allows to replace sftp packet read and write with
completely asynchronous versions needed to properly hook in existing
ssh_terminal implementation.

Issue #4258
 2021-10-13 14:01:02 +02:00
Tomasz Gajewski
e34d1550a4 ssh_terminal: added missing include 
Issue #4258
 2021-10-13 14:01:02 +02:00
Christian Helmuth
77a5cf7fd4 vbox6: enable audio via OSS plugin 2021-10-13 14:01:02 +02:00
Christian Helmuth
2ba5e4a5b1 libc/oss: check argp per ioctl 
... as some ioctls do not require a valid pointer.
 2021-10-13 14:01:02 +02:00
Christian Helmuth
5c82045170 vbox6: enable mouse wheel reporting 2021-10-13 14:01:02 +02:00
Christian Helmuth
4a87fcc4cf vbox6: fix TPR handling 2021-10-13 14:01:02 +02:00
Christian Helmuth
e5b828ae8f Switch to genodelabs depot in depot_download test 2021-10-13 14:01:02 +02:00
Martin Stein
f1b72d0281 cxx: define delete operator with alignm. arg 
Using 'alignas' in declarations might cause GCC to request for an
implementation of 'operator delete(void*, unsigned long, std::align_val_t)'
although it might actually never be called. This commit adds a dummy
implementation to 'cxx/new_delete.cc' that does nothing more than printing an
error to the log that a proper implementation is missing. This approach is
coherent with our treatment of other global delete operators.

Ref #4217
 2021-10-13 14:01:02 +02:00
Martin Stein
ebd140cacb reconstructible: respect alignment of payload 
If one has an object X that has a minimum alignment requirement specified
through 'alignas' this requirement is normally inherited by objects that have
object X as member, and by those that have objects as member that have X as
member, and so on... . However, this chain used to get silently interrupted
(dropping the minimum alignment requirement to 8 again) at objects that are
managed with Genode::Reconstructible or Genode::Constructible. In order to fix
this, the commit ensures that Genode::Reconstructible (and therefore also
Genode::Constructible) has at least the minimum alignment requirement (using
'alignas') as the object it manages.

Ref #4217
 2021-10-13 13:59:57 +02:00
Christian Helmuth
64e2912a2f version: 21.08 2021-08-31 12:02:22 +02:00
Norman Feske
339dda8b43 News item for version 21.08 2021-08-31 12:02:22 +02:00
Norman Feske
397a3e45d1 Release notes for version 21.08 2021-08-31 12:02:22 +02:00
Norman Feske
f8898f3a56 doc: add server/black_hole to components.txt 2021-08-31 12:02:22 +02:00
Christian Helmuth
83c5648d33 depot: update recipe hashes 2021-08-30 15:00:39 +02:00
Martin Stein
7fdebc6a09 nic_router: let DHCP client request domain name 
The NIC router parses, stores and forwards DNS domain names from DHCP replies.
Yet the routers DHCP client used to not request DNS domain-name information on
DHCP requests. This caused DHCP servers to skip this information on their
replies although it was available. This commit fixes the issue by adding the
DNS domain name code to the request parameter list of requests from the routers
DHCP client.
 2021-08-30 15:00:38 +02:00
Sebastian Sumpf
4782f9376c recipe: Qt5 texedit adjust to new Mesa version 
Add dependency to 'libdrm' as it is required by Mesa 21

issue #4254
 2021-08-30 15:00:38 +02:00
Christian Helmuth
a362505e8a usb_host: report bInterfaceProtocol too 
Thanks to Raphael for the patch.
 2021-08-30 15:00:38 +02:00
Christian Prochaska
ece837e8b8 qt5: prepare QtWebEngine sndio backend 
Fixes #4257
 2021-08-30 15:00:38 +02:00
Christian Prochaska
eedbcf88ec os: add black hole component 
The 'black_hole' component provides dummy implementations of common
session interfaces.

At this time, only the 'Audio_out' session is provided if enabled
in the configuration of the component:

<config>
	<audio_out/>
</config>

Issue #3653
 2021-08-30 15:00:38 +02:00
Johannes Schlatow
2b0bb6dda0 trace: comment x86 timestamp() implementations 
Fixes genodelabs/genode#4243
 2021-08-30 15:00:38 +02:00
Sebastian Sumpf
31b049864c qt5_opengl: Disable Vulkan feature 
With Mesa-21, Qt5 detects Vulkan to be enabled, even though it is not
supported in Genode. Also add DRM library in Qt5 run script.

issue #4254
 2021-08-30 15:00:38 +02:00
Sebastian Sumpf
46be4f1145 mesa_demo: adjust to Mesa 21.0.0 
* support argc/argv + double buffering (Alexander Boettcher)
* Route intel/gpu driver to platform driver
* Use Gui session instead of framebuffer session (Josef Söntgen)

issue #4254
 2021-08-30 15:00:38 +02:00
Sebastian Sumpf
e4ae817e82 ram_fb_drv: RAM framebuffer driver for Qemu 
Enable "-device ramfb" to use in Qemu. Also add drivers interactive
using this framebuffer for the "virt_qemu" platform.

issue #4254
 2021-08-30 15:00:38 +02:00
Sebastian Sumpf
38d731bd79 libports: Add Mesa-21.0.0 support 
* Switch mesa support from DRI to gallium
  Supported drivers are
    - softpipe (Sebstian Sumpf)
    - iris for Intel GPUs (Alexander Boetcher)
    - etnaviv for Vivante GPUs (Josef Söntgen)

* Mesa's generated files are placed into 'contrib/mesa-<hash>/generated'
  and are cloned per default from a separate Git repo in order to avoid
  hash updates upon package build. In case you need to generate files
  yourself use
  ! prepare_port mesa GENERATE_FILES=1

issue #4254
 2021-08-30 15:00:38 +02:00
Sebastian Sumpf
9041567f14 libports: Remove Mesa-11.2.2 
issue #4254
 2021-08-30 15:00:38 +02:00
Sebastian Sumpf
b6ec4bdf05 libc/stdcxx ABI: symbols for Mesa-21.0.0 
issue #4254
 2021-08-30 15:00:38 +02:00
Josef Söntgen
dd90424129 libports: libdrm-2.4.105 update 
Support for iris and etnvaviv

* entaviv:
  - libdrm on FreeBSD is not prepared for !PCI (and libc our is missing
    <sys/pciio.h>
  - missing <sys/types.h> include in xf86drmMode.c
  - etnaviv relies on linux header files - dummy in $(INC_DIR)
  - IOCTL FreeBSD ↔ Linux have swapped IO/OUT bit
  - O_CLOEXEC differs between FreeBSD ↔ Linux

issue #4254
 2021-08-30 15:00:38 +02:00
Sebastian Sumpf
c26c50e59b libports: remove libdrm 2.4.21 
issue #4254
 2021-08-30 15:00:38 +02:00
Josef Söntgen
ac9e0947fd libc: add HM_REALMEM sysctl 
Used by Mesa in the Gallium etnaviv driver.

issue #4254
 2021-08-30 15:00:37 +02:00
Josef Söntgen
fb4d357b59 libc: pull in sys/pciio.h 
This header is needed by recent versions of libdrm that feature
FreeBSD support.

issue #4254
 2021-08-30 15:00:37 +02:00
Alexander Boettcher
7a4626861e gpu/intel: add w/a for gen9 kabylake a0-b0 stepping 
issue #4254
 2021-08-30 15:00:37 +02:00
Alexander Boettcher
bac7381be3 gpu/intel: adjust submit batch to linux 5.13 handling 
issue #4254
 2021-08-30 15:00:37 +02:00
Alexander Boettcher
6b1f1794c4 gpu/intel: add gen9 forcewake support 
issue #4254
 2021-08-30 15:00:37 +02:00
Alexander Boettcher
eab92f8d6f gpu/intel: add clock gating for kabylake 
issue #4254
 2021-08-30 15:00:37 +02:00
Alexander Boettcher
ee283c0d12 gpu/intel: dump more in error case 
- show hardware read tail & head pointer of ring buffer

issue #4254
 2021-08-30 15:00:37 +02:00
Alexander Boettcher
c5d8a43418 gpu/intel: sanity check tail pointer alignment 
According to spec the tail pointer points to the next qword instructions
which will be used by the software.

p 1354, Doc Ref # IHD-OS-BDW-Vol 2c-11.15

issue #4254
 2021-08-30 15:00:37 +02:00
Alexander Boettcher
cc2363d421 gpu/intel: report slice, subslice, eu via gpu info 
used by libdrm & mesa-21 clients

issue #4254
 2021-08-30 15:00:37 +02:00
Alexander Boettcher
b287c4888a gpu/intel: update context inhibit setup 
issue #4254
 2021-08-30 15:00:37 +02:00
Alexander Boettcher
66ac2dc635 gpu/intel: avoid one-by-off in ring buffer 
during wrap.

issue #4254
 2021-08-30 15:00:37 +02:00
Alexander Boettcher
6c7bcdd32e gpu/intel: add gen9 w/a for submit batch 
issue #4254
 2021-08-30 15:00:37 +02:00
Alexander Boettcher
b2440a72c3 gpu/intel: add more debug registers 
issue #4254
 2021-08-30 15:00:37 +02:00
Alexander Boettcher
8a285a7bee gpu/intel: add more debug registers 
issue #4254
 2021-08-30 15:00:37 +02:00
Alexander Boettcher
c9809fde67 gpu/intel: remove unused superpage scratch code 
Superpages (2M, 1G) are not supported by now, but partially copied over code
from base-hw was around. Remove unused register definitions and remove
non-working super page code do avoid confusion.

issue #4254
 2021-08-30 15:00:37 +02:00
Alexander Boettcher
de8327c11a gpu/intel: add more supported chips 
support for Gen > 8

+ determine platform, revision and stepping

issue #4254
 2021-08-30 15:00:37 +02:00
Alexander Boettcher
0aa17661cc gpu/intel: avoid overlapping ggtt range allocation 
Size argument of ggtt free range check is ignored, which leads to
overlapping allocations inside, which leads to unavailable IO-MEM exceptions
thrown by core.

issue #4254
 2021-08-30 15:00:37 +02:00
Norman Feske
82a51d8eaa build: operate ccache in preprocessor mode 
This avoids consistency problems when header files are shadowed.

Fixes #4256
 2021-08-30 15:00:37 +02:00
Norman Feske
040628894c build: introduce GLOBAL_DEPS variable 
The new varible can be used to trigger the rebuild of the entire target
whenever any of the listed files changes.

Fixes #4255
 2021-08-30 15:00:37 +02:00
Johannes Schlatow
b5dd1dd01b trace: consolidate Trace_buffer implementations 
Both, trace_logger and vfs_trace had their own trace_buffer.h. This
commit consolidates the existing implementations and provides the
resulting trace_buffer.h at 'include/trace/'. It thereby becomes part of
the trace api archive.

genodelabs/genode#4244
 2021-08-30 15:00:37 +02:00
Johannes Schlatow
ffbd26d63f depot: add trace policy api 
genodelabs/genode#4244
 2021-08-30 15:00:36 +02:00
Johannes Schlatow
1416b2258f Trace_buffer: do not reprocess last read entry 
genodelabs/genode#4244
 2021-08-30 15:00:36 +02:00
Norman Feske
436d946300 recipes/api/genode_c_api: add src/lib/genode_c_api 
The source codes are needed to use the API because they are statically
linked.
 2021-08-30 15:00:36 +02:00
Norman Feske
324ad33736 lx_emul: prevent clk_disable_unused 
Issue #4253
 2021-08-30 15:00:36 +02:00
Norman Feske
2cb4157211 lx_emul: support IRQCHIP_DECLARE initcalls 
Similar to CLK_OF_DECLARE, this patch extends lx_emul with ability to
extend the __irqchip_of_table with secondary interrupt controllers.

Issue #4253
 2021-08-30 15:00:36 +02:00
Norman Feske
c95af254f4 lx_emul: sanitize size 0 for __kmalloc 
Driver code such as mfd-core.c may pass 0 as argument n to kcalloc,
which eventually results in an allocation size 0.

  res = kcalloc(cell->num_resources, sizeof(*res), GFP_KERNEL);

Since 'res' is checked against NULL for success, kmalloc must not return
a NULL pointer in this case. The patch works around this issue by
forcing an allocation size of 1 byte in this case.

Issue #4253
 2021-08-30 15:00:36 +02:00
Norman Feske
29032caf40 lx_emul: support CLK_OF_DECLARE initcall mechanism 
Clock providers such as drivers/clk/sunxi-ng/ccu-sun8i-r.c don't use
regular init calls but declare their init functions via CLK_OF_DECLARE,
which fill the __clk_of_table. Linux populates the table statically by
using special sections declared in the linker script. In contrast, we
populate the table by expanding the macro to global constructor
functions.

The __clk_of_table is then processed by the call of of_clk_init(NULL).

Issue #4253
 2021-08-30 15:00:36 +02:00
Norman Feske
36af114d78 lx_emul: enhance krealloc implementation 
Issue #4253
 2021-08-30 15:00:36 +02:00
Christian Prochaska
149bd999f3 vfs_oss: support more commands for VirtualBox 6 and sndio 
Fixes #4252
 2021-08-30 15:00:36 +02:00
Christian Helmuth
a6fb61dbf2 vbox6: support dynamic framebuffer resolution 
Issue #4031
 2021-08-30 15:00:36 +02:00
Johannes Schlatow
50cc51f132 tool: support new u-boot FIT image format 
Fixes genodelabs/genode#4250
 2021-08-30 15:00:36 +02:00
Christian Helmuth
c54473abea vbox6: update to version 6.1.26 
Issue #4031
 2021-08-30 15:00:35 +02:00
Christian Helmuth
611e93a5f2 vbox6: kick AHCI asynchronous IO thread 
Issue #4031
 2021-08-30 15:00:35 +02:00
Christian Helmuth
d6bde82894 vbox6: add libyuv to pkg recipe 
Issue #4031
 2021-08-30 15:00:35 +02:00
Johannes Schlatow
88b3880c77 base: fix shift overflow error in Bitfield::mask() 
Fixes genodelabs/genode#4249
 2021-08-30 15:00:35 +02:00
Christian Helmuth
7618c9410a depot: update recipe hashes 2021-08-18 15:06:12 +02:00
Johannes Schlatow
5e284bfb35 trace: disable tracing when owner disappears 
* Disable trace source and release ownership on subject destruction.
* Note, since the policy module is also destroyed on descruction of the
  session component, the traced component must not access the policy
  module when acknowledging the disabled state (else: page fault).

Fixes genodelabs/genode#4247
 2021-08-18 15:06:12 +02:00
Johannes Schlatow
81d939f947 test-trace: test destruction of trace subjects 
If the trace subjects are not properly destructed when the TRACE client
disappears, enabled sources will be owned by a non-existing client.
In other words, when a TRACE client disappears all sources owned by the
client must be disabled.

genodelabs/genode#4247
 2021-08-18 15:06:12 +02:00
Johannes Schlatow
9898341d4a depot_autopilot: fix ineffective test-trace 
test-trace always passed, although tracing was never enabled because the
trace subject was not within the first 32 subjects.

* increase number of queried subjects
* output error if trace subject was not found

genodelabs/genode#4247
 2021-08-18 15:06:12 +02:00
Martin Stein
812fdec27c nic_router: propagate dns domain name via dhcp 
With this commit, the NIC router DHCP client reads out the first DNS domain
name (DHCP option 15) if any from a DHCP reply that generates an IPv4 config
for a domain and stores the name together with the IPv4 config for that domain.
DNS domain names are reported via the new report tag '<dns-domain>' if the
'config' attribute in the config tag '<report>' is set.

Furthermore, the NIC router DHCP server becomes able to obtain a DNS domain
name from another domain that has a DHCP client dynamically (given the config
attribute 'dns_config_from' is set and no static DNS config is given) or
statically from its configuration (new config tag '<dns-domain>') and propagate
this name with DHCP replies (DHCP option 15).

The 'nic_router_dhcp_*' tests are adapted to test the new feautures.

The commit also gets rid of some mirrored files in
'test/nic_router_dhcp/manager'.

Fixes #4246
 2021-08-18 15:06:12 +02:00
Martin Stein
fa64aae7f8 nic_router: config attribute 'dns_config_from' 
WARNING: BREAKS CONFIG COMPATIBILITY!

This commit changes the configuration interface of the NIC router in a way that
may break systems that use the component without proper adjustment!

How to adjust:
At each occurrence of the 'dns_server_from' attribute in a NIC router
configuration replace the attribute name with 'dns_config_from'. The attribute
value remains unaltered.

DETAILED DESCRIPTION

The new attribute name 'dns_config_from' reflects that also other aspects of
the DNS configuration of the denominated domain are used by the DHCP server
that holds the attribute. This commit is a preparation for forwarding also the
domain name (DHCP option 15) with the mechanism behind the attribute.

Ref #4246
 2021-08-18 15:06:12 +02:00
Martin Stein
1111472af7 nic_router: make Ipv4_config a class 
The fact that the IPv4 config was a struct with all data members public was a
mere leftover of an early state of the NIC router. Today, the router
implementation style is to avoid structs and public data members wherever
possible.

This commit slightly changes the behavior of the router regarding log output.
The router used to print malformed IPv4 configurations to the log only if
the 'verbose' config flag was set using this style:

! [my_domain] malformed dynamic IP config: interface 10.0.2.1/24 ...

Now, malformed IPv4 configurations are only printed if the
'verbose_domain_state' config flag is set (like with any IP4v configuration
states) using this style:

! [my_domain] dynamic IP config: malformed (interface 10.0.2.1/24 ...)

Fixes #4242
 2021-08-18 15:06:12 +02:00
Martin Stein
9e6f7988c2 nic_router: list dns servers in one dhcp option 6 
The NIC router DHCP server used to add an extra option 6 field to DHCP replies
for each DNS server address. This conflicts with RFC #2132 section 3.8 which
states that the addresses should be listed within one option 6 field without
delimiter. The discrepancy is fixed by this commit.

Ref #4242
 2021-08-18 15:06:12 +02:00
Johannes Schlatow
80c1459e79 fs_query: catch File::Truncated_during_read 
Fixes genodelabs/genode#4241
 2021-08-18 15:06:12 +02:00
Johannes Schlatow
0840cfe834 vfs_trace: report correct size for value files 
File size must be the same as the number of bytes that can be read from
the file. Otherwise, this will trigger a `Truncated_during_read`
exception.

Fixes genodelabs/genode#4240
 2021-08-18 15:06:12 +02:00
Martin Stein
e648e7255a nic_router: optional icmp type 3 on fragm. ipv4 
Via a new configuration attribute, the user can decide whether the router
should answer dropped fragmented IPv4 with an ICMP "destination unreachable"
packet and, if so, which value the ICMP code field of this packet should have.
The default is that the router doesn't send such responses (silently dropping
fragmented IPv4). The behavior is tested by the 'nic_router_ipv4_fragm' test.

Fixes #4236
 2021-08-18 15:06:12 +02:00
Martin Stein
06a4608f4a nic_router: report dropped fragmented IPv4 
If the new attribute 'dropped_fragm_ipv4' of the <report> tag in the NIC router
config is set "yes", the router will report the number of packets that were
dropped per interface respectively domain because fragmented IPv4 is not
supported. The default is not to report the counter. The behavior is tested by
the 'nic_router_ipv4_fragm' test.

Ref #4236
 2021-08-18 15:06:12 +02:00
Martin Stein
619474bc90 nic_router: drop fragmented IPv4 
The NIC router used to ignore the IPv4 header fields "More fragments" and
"Fragment offset" completely. Therefore higher-level protocols of fragmented
IPv4 were interpreted wrong because each fragment was considered a self-
standing packet, expecting, for instance UDP/TCP headers somewhere inside of
the UDP/TCP data field. Normally, such packets were dropped as soon as the
UDP/TCP checksum check failed because of the misinterpretation. However,
it was also possible for fragmented IPv4 to pass the router although normally
only partially.

IPv4 fragmentation support in the router would introduce some potential
security risks and is presumably not an easy endeavor. So, for now, we settled
on not supporting IPv4 fragmentation. With this commit, the router simply drops
all fragmented IPv4. This is reflected to the log for each fragment as "drop
packet (fragmented IPv4 not supported)" when 'verbose_packet_drop="yes"' is
configured.

The new test 'run/nic_router_ipv4_fragm' is an automated test for this
behavior. The test is added to the autopilot list.

Ref #4236
 2021-08-18 15:06:12 +02:00
Martin Stein
b0e558f486 net/icmp: cast from integer to 'Code' enum 
Add a function to the header for doing this cast.

Ref #4236
 2021-08-18 15:06:12 +02:00
Christian Prochaska
d7a27c448d qt5: implement 'CancelableSyncSocket::Shutdown()' 
Fixes #4239
 2021-08-18 15:06:12 +02:00
Martin Stein
626b2f9cf2 run/libc_vfs_fs_fat: add to autopilot 
Excludes board 'virt_qemu' from the test as we don't have block device drivers
for this board.

Ref #4220
 2021-08-18 15:06:12 +02:00
Martin Stein
2533d7b4b6 vfs/fatfs: cleanup implementation 
- remove redundant file system factory

- remove dead code block

  The code was guarded by preprocessor directives checking whether the
  contrib code define "_USE_MKFS" is 1. As "_USE_MKFS" is not set one
  for our port of FAT, the code was never executed and can be removed.

- remove uneffective config attributes

  Apparently, the former XML attributes to the plugin 'drive' and
  'codepage' had no effect. I tested them in a scenario with the VFS
  block server on a disk-image boot-module as back end. Regardless of
  the 'drive' value, the block session label was always "0". Regardless
  of the 'codepage' value, the FAT on the disk image succeeded to mount
  when not using '--codepage' for 'mkfs.fat' and failed to mount when
  using '--codepage' to specify a supported but foreign codepage for
  'mkfs.fat' (e.g. "720").

Ref #4220
 2021-08-18 15:06:12 +02:00
Martin Stein
60c8369718 lib/fatfs: get rid of global static constructors 
There was one global static constructor:

! namespace Fatfs { static Constructible<Platform> _platform; }

This caused applications that used the lib or the <fatfs> VFS plugin to end up
in an uncaught exception due to Genode::Component complaining that method
'construct' returned without executing pending static constructors if they
didn't call Genode::Env::exec_static_constructors().

As the use of Genode::Env::exec_static_constructors() is discouraged in Genode,
this commit rather moves the '_platform' object to the scope of the
initializing function and introduces a global static pointer to the object that
gets set by the initializing function. Although this prevents the exception, it
is, technically speaking even worse than the former solution as the new pointer
isn't checked for validity in contrast to the 'Constructible' object.

However, so far, I don't see a clean solution to this problem without the need
for Genode::Env::exec_static_constructors().

Fixes #4220
 2021-08-18 15:06:12 +02:00
Christian Helmuth
b59e2ba677 microcode_intel: archive moved to GitHub finally 
As version 20180807a is not provided on GitHub, I updated the port to
the most recent release 20210608.

Issue #2762
 2021-08-18 15:06:12 +02:00
Sebastian Sumpf
c3e8c22a6d sculpt: integrate intel_gpu_drv 
* the GPU multiplexer now offers the platform service to the Intel
  framebuffer driver (driver_manager)
* ajdusted drivers_managed-pc to hand out resources to the GPU driver
* adjust quotas

issue #4233
 2021-08-18 15:06:11 +02:00
Sebastian Sumpf
2fe70f111b intel_fb_drv: Aperture exhaustion and display IRQ patch 
* display an error in case the aperture is too small for all
  framebuffers
* handle display engine IRQs only

issue #4233
 2021-08-18 15:06:11 +02:00
Sebastian Sumpf
1727de30b7 gpu/intel: free DMA, clear ggtt 
* free DMA caps in case platform client's session is closed
* clear GGTT of platform client upon session close

issue #4233
 2021-08-18 15:06:11 +02:00
Sebastian Sumpf
6c003a13d2 gpu/intel: add platform service for display handling 
The platform services is intented to be used by dde_linux's intel_fb_drv
in order to initlialize displays.

* implement and announce platform session
* limit accessible GTT and aperture of client to 64 MB
* forward display engine IRQs to platform client
* move all PCI resources to 'Igd::Resources' class in order to make them
  accessible by the platform service and the GPU driver
* fix fence register allocation for id zero (return true)

issue #4233
 2021-08-18 15:06:11 +02:00
Alexander Boettcher
11192b18e6 gpu/intel: dump status context register bits 
Issue #4148 #4233
 2021-08-18 15:06:11 +02:00
Alexander Boettcher
fe867765a8 gpu/intel: support multiple exec buffer per client 
required for mesa-21

Issue #4148 #4233
 2021-08-18 15:06:11 +02:00
Alexander Boettcher
ffc89f3edf gpu/intel: let guard page handling up to client 
For mesa-21 the client takes care and manages
the virtual address space of the vGPU by itself and the intel/gpu driver
can't add silently a guard page anymore. Move the patch to the drm/ioctl
of the former mesa version.

Issue #4148 #4233
 2021-08-18 15:06:11 +02:00
Alexander Boettcher
067b7d7c67 gpu/intel: deny virtual address above 1 << 48 
commit avoids memory corruption in the ppgtt page insertion code

Issue #4148 #4233
 2021-08-18 15:06:11 +02:00
Josef Söntgen
4b653fbac1 gpu/intel: handle double insertion in PPGTT 
As the PPGTT is populated by the client, the client has to account for
guard pages and so forth.

Issue #4148 #4233
 2021-08-18 15:06:11 +02:00
Josef Söntgen
7dc997c8e6 gpu/intel: handle CSB correctly 
As it turns out reading both dwords as qword results in mirrored
values, DDTT.

Issue #4148 #4233
 2021-08-18 15:06:11 +02:00
Alexander Boettcher
5d6ea5ef22 gpu/intel: allocate vcpu/engine as part of session 
Issue #4148 #4233
 2021-08-18 15:06:11 +02:00
Alexander Boettcher
a721933771 gpu/intel: move context/ring handling into objects 
Issue #4148 #4233
 2021-08-18 15:06:11 +02:00
Alexander Boettcher
c949e5c90d gpu/intel: remove allocator_guard 
Issue #3759 #4148 #4233
 2021-08-18 15:06:11 +02:00
Alexander Boettcher
243a9ec3ca gpu/intel: free up context/memory ring 
_unmap_dataspace_ggtt requires the cap of Ggtt::Mapping (ring_map, ctx_map)
in order to find the right metadata and to free up the ggtt entries. Also the
pte range is removed already if the metadata was found.

Issue #4148 #4233
 2021-08-18 15:06:11 +02:00
Alexander Boettcher
b3147050cc gpu/intel: add support to report finished batches 
- extend gpu multiplexer to report last completed execution buffer
- on client side lookup finished buffers and disable busy state

issue #4233
 2021-08-18 15:06:11 +02:00
Martin Stein
f8953de7ac nic_router: replace term "uplink" with "NIC client" 
BREAKS CONFIG COMPATIBILITY:
This commit changes the configuration interface of the NIC router in a way that
may break systems that use the component without proper adjustment!

HOW TO ADJUST:
At each occurrence of the '<uplink ...>' tag in a NIC router configuration
replace the tag name 'uplink' with 'nic-client'. The rest of the tag stays the
same.

The term "uplink" for network interfaces in the router that have a NIC session
client as back end was introduced in a time when Uplink sessions didn't yet
exist. Now, they do and, although both an uplink and an Uplink session
normally describe a network session between router and network device driver,
they are based on two different service types (NIC and Uplink). This can easily
cause confusion when integrating the router (the <uplink> is not related to
Uplink sessions) or trying to understand its functioning (an 'Uplink' object
has nothing to do with the Uplink service).

Therefore, this commit introduces the more specific term "NIC client" for an
interface that is based on a NIC session requested by the router. This doesn't
imply any semantic changes at the NIC router. However, the commit also brings a
broader update of the router's README and removes the term "downlink" that was
used only in documentation to refer to interfaces backed by a NIC session
provided by the router. The term was only associated with this meaning because
it is the natural counterpart to an uplink. This isn't appropriate anymore as
the terms for interface types have moved to a more technical level.

The commit adjusts all scenarios in the basic Genode repositories properly.

Fixes #4238
 2021-08-18 15:06:11 +02:00
Christian Helmuth
fce525f122 depot: update recipe hashes 2021-07-28 11:27:05 +02:00
Martin Stein
2afae7e7c1 nic_router: uncaught exc. on handling link state 
An interface that received a signal for a link-state change accessed its
domain reference without assuming that it could not be attached to a domain
at that moment. This caused the NIC router to crash with an uncaught exception
of type 'Net::Pointer<Net::Domain>::Invalid'. The commit adds a catch
directive for this exception resulting in the handler doing nothing if not
attached to any domain.

Fixes #4222
 2021-07-28 11:27:05 +02:00
Martin Stein
d06773b957 run/tz_vmm: remove usb_armory support 
The test script failed during preparation of the on-target execution for
USB Armory with the following error:

! can't read "tz_vmm_block_irq": no such variable

Presumably, the script wasn't run anymore since the introduction of the
'tz_vmm_block_irq' variable for i.MX53 QSB. As we do not have infrastructure
for automated testing of the USB Armory and there seems to be not much
interest in using Genode on this platform, this commit simply removes the
support from the script.
 2021-07-28 11:27:05 +02:00
Martin Stein
c8a8cbd7be run/tz_vmm: fix deprecated use of board specs 
Filtering boards in a run script by specs isn't the right way anymore (the
specs do not exist anymore). Nowadays, we have to use [have_board] instead.

Ref #4229
 2021-07-28 11:27:05 +02:00
Martin Stein
00e8e363d8 run/tz_vmm: fix broken network test 
For unknown reasons, the former 'wget genode.org' call, that was meant to test
network in the Trustzone guest on imx53_qsb_tz, didn't succeed anymore although
the same call succeeded on my Sculpt VM Linux. However, 'ping 1.1.1.1' still
works, so, the script now uses this as test for networking instead.

Fixes #4229
 2021-07-28 11:27:05 +02:00
Martin Stein
8bc861ca71 nic_router: do not re-use ARP request as reply 
So far, in order to create an ARP reply, the NIC router merely created a copy
of the corresponding ARP request and modified only those values that differ.
This approach has the disadvantage of re-using bad parameters from a broken
request. The specific use-case that made this visible was an early version of
the Pine board network driver that used to forward ARP requests with a greater
size than required. The ARP replies of the router re-used this size and
confused other network nodes with that. In general, the NIC router should
rely on the data of incoming packets the least possible. Therefore, with this
commit, the router creates a new ARP reply from scratch and uses only those
values required from the corresponding ARP request.

Fixes #4235
 2021-07-28 11:27:04 +02:00
Martin Stein
9384e075cb ipv4: get rid of C++ bitsets, add DF/MF flags 
The former declaration of the IPv4 packet did not only use the questionable
tool of implementation-defined C++ bitsets but also lacked access to flags
"don't fragment" (DF) and "more fragments" (MF). This commit replaces the
C++ bitsets by using the register framework and introduces accessors for the
missing flags.

Ref #4236
 2021-07-28 11:27:04 +02:00
rolandbaer
52011ec034 smbios_decoder: fix timer error message 
added timer to run file to fix error message.

Fixes #4237
 2021-07-28 11:27:04 +02:00
Alexander Boettcher
908d581a8c qemu-usb/webcam: reset delay packet state 
on feature reset initiated by guest

Issue #4078
 2021-07-28 11:27:04 +02:00
Johannes Schlatow
b38ec9f238 usb_webcam: correctly enable auto exposure mode 
genodelabs/genode#4231
 2021-07-28 11:27:04 +02:00
Stefan Kalkowski
9334ec09e2 sculpt: route non-usb block sessions to 'drivers' 2021-07-28 11:27:04 +02:00
Johannes Schlatow
1bfc828826 libuvc: add UVC 1.50 support 
genodelabs/genode#4231
 2021-07-28 11:27:04 +02:00
Johannes Schlatow
b51b9e1ef3 libusb: fix size of control messages 
Fixes genodelabs/genode#4230
 2021-07-28 11:27:04 +02:00
Norman Feske
3d36291d7f run/load/fastboot: support 64-bit ARM 
This patch lifts the limitation for 32-bit ARM platforms and makes the
parameter --load-fastboot-device optional. If only one device is
present, it can be omitted.

Fixes #4232
 2021-07-28 11:27:04 +02:00
Johannes Schlatow
2afb7c5567 nic_router: fix Tcp_packet header fields 
data_offset and NS flag were incorrectly read due to missing endianess
conversion. Also fix name of CWR flag.

Fixes genodelabs/genode#4227

genodelabs/genode#4227
 2021-07-28 11:27:04 +02:00
Stefan Kalkowski
ee045a68cc os: introduce C-API to Genode services 
This commit introduces a C-API to the Uplink session, as well as to
serve as a Block service. It can be used by drivers ported from
C-only projects, like the Linux kernel, or BSD kernels for instance.

Fix #4226
 2021-07-28 11:27:04 +02:00
Stefan Kalkowski
1a526e73a3 dde_linux: introduce new lx_emul/lx_kit library 
The re-newed approach currently supports ARM 64-bit only.
It depends on the Platform API of the ARM architecture.
It tries to meet the original semantic of the Linux kernel
functions as far as possible. To achieve this, device drivers
using this library should reference the original Linux kernel
headers at foremost. Only the headers in `src/include/lx_emul/shadow`
have to shadow clone the original ones.

Fix #4225
 2021-07-28 11:27:04 +02:00
Johannes Schlatow
1aba330ae6 dde_linux: fix packet size calc. of Nic drivers 
skb_push() already increases the skb->len by ETH_HLEN, hence adding
ETH_HLEN to the packet_size is redundant.

A too large packet size becomes a problem for large MTUs. With a maximum
MTU of 1500, adding ETH_HLEN twice will lead to a packet size of 1528.
Since this is larger than what we expect for good-old Ethernet (max. 1522),
some clients (e.g. the e1000 model in vbox5) may drop these packets.

Fixes genodelabs/genode#4228
 2021-07-28 11:27:04 +02:00
Christian Helmuth
119d72ad94 fetchurl: User-Agent header and HTTP response code 
I discovered thinkbroadband.com requires the User-Agent header field and
rejects requests missing it with HTTP response code 403 "access to the
requested resource is forbidden". Now, fetchurl always adds the
User-Agent header fetchurl/LIBCURL_VERSION.

Also the error message now contains the HTTP response code.
 2021-07-28 11:25:21 +02:00
Stefan Kalkowski
b16bb82f8b dde_linux: mark lx_emul and lx_kit as legacy 
* Move headers in src/include into legacy sub-directory
* Move src/lx_kit to src/lib/legacy/

Fix #4223
 2021-07-28 11:25:21 +02:00
Stefan Kalkowski
f939b9ffb5 base-hw: print addr as hex value in cache warning 
Ref #4207
 2021-07-28 11:25:21 +02:00
Norman Feske
45f5ed173a tool/run: use [board] for naming driver packages 
This patch replaces the enumeration of boards in the drivers_nic_pkg and
drivers_interactive_pkg functions by inferring the package names from
the [board] name. This eases the addition of driver packages in external
repositories.

Fixes #4221
 2021-07-28 11:25:21 +02:00
Norman Feske
6de19e4a9b tool_chain_auto.run: exercise 'ln -s' 
as a regression test for issue #4219.
 2021-07-28 11:25:21 +02:00
Norman Feske
5138aeba80 libc: fix nested monitor call in 'symlink' 
The symlink implementation wrongly constructed a 'Sync' object within
the context of a monitor call. The 'Sync' constructor indirectly
depended on libc I/O for obtaining the current time, ultimately
resulting in a nested attempt of a monitor call. This could be
reproduced via the base.run script:

  $ cd /home
  $ ln -s a b

The 'ln' command resulted in the following log message:

  [init -> /bin/bash -> 7] Error: deadlock ahead, mutex=0x10ff8c70, return ip=0x500583a7

The patch fixes the problem by splitting the single monitor call into
two monitor calls and moving the construction of the 'Sync' object
in-between both monitor calls, thereby executing the constructor at the
libc application level.

Fixes #4219
 2021-07-28 11:25:21 +02:00
Norman Feske
f3908b8283 tool: refine help text of list_dependencies 
Issue #4204
 2021-07-28 11:25:21 +02:00
Christian Helmuth
fdc4bd2f90 tool/run: absolute source path for load_tftp_inst_cmd 2021-07-28 11:25:21 +02:00
Sebastian Sumpf
4d4cc4fd02 libm: use libm from libc and remove openlibm 
- enable libm for x86/arm
- update ieee754 to slightly changed output
- remove openlibm from port

fixes #4213
 2021-07-28 11:25:21 +02:00
Sebastian Sumpf
4b10aa94ec base-hw: save/restore fpcr register for ARM_V8 
The floating-point (fpcr) control register is user land accessible and
controls, for example, rounding mode.

fixes #3723 and related to #4213
 2021-07-28 11:25:20 +02:00
Martin Stein
688379d1ed sel4: fix segmentation fault with CCACHE=yes 
Building the elfloader in kernel-sel4.inc has a problem with Genodes CCACHE
make variable. When issuing ...

! ./tool/depot/create mstein/bin/*/base-sel4-* CCACHE=yes

..., building the elfloader used to consume all memory of the host system and
then run into a segmentation fault:

! make[6]: *** [elfloader/elfloader.o] Segmentation fault (core dumped)

This is because the other build system invokes the CCACHE variable as a command
in front of the compiler command. If CCACHE is set to 'yes', the 'yes' command
is called and produces an endless output into some output file. The problem
can be fixed by locally re-setting the CCACHE variable for the
'make ... elfloader' command to 'ccache' (Genode CCACHE==yes) or '' (Genode
CCACHE!=yes).

Fixes #4212
 2021-07-28 11:25:20 +02:00
Martin Stein
0074a7c4ac tool/depot: fix forwarding of CCACHE 
The depot tool 'build_bin_archive' used to ask only whether CCACHE is empty or
not and, if it wasn't empty, the tool added 'CCACHE := yes' to the build.conf
of the temporary build dir. This caused the sub-build processes to use ccache
even when the top-level CCACHE variable was set to a bogus value.

Also adds documentation of the CCACHE variable to the help message of the
'build' and 'build_bin_archive' tool.

Ref #4212
 2021-07-28 11:24:56 +02:00
Martin Stein
388e2a0e6d fec nic driver: fix compiler warnings 
Fixes #4178
 2021-07-28 11:24:55 +02:00
Christian Prochaska
a856bfb4ab qt5: implement 'SyncSocket' with pipes on Genode 
Fixes #4208
 2021-07-28 11:24:55 +02:00
Christian Prochaska
3824c0ca5f qt5: ignore 'fcntl(F_SETLK)' error in sqlite3 implementation 
Fixes #4194
 2021-07-28 11:24:55 +02:00
Robin Eklind
40e2aa6617 builddir/run_x86: add note regarding missing -no-kvm switch 
Also, add -enable-kvm notes as commented out QEMU_OPT option.

Updates #500.
 2021-06-28 11:38:20 +02:00
Martin Stein
2d017ad7b7 nic_router: guard against exception in free_ip 
Adds try-catch-statement with diagnostic errors in Dhcp_server::free_ip in
order to guard against exceptions from the underlying bit allocator. These
exceptions should never happen given that the router is programmed correctly
and always feeds Dhcp_server::free_ip with sane arguments (which it should).
However, should this not be the case, we can assume that the failed IP freeing
indicates that the IP isn't allocated anyway and it's fine to continue using
the router. Furthermore, IP allocations are a mere client service and not
relevant for the integrity or safety of the router.

Ref #4200
 2021-06-25 11:57:28 +02:00
Martin Stein
be644098d7 nic_router: fix exc. in Interface::handle_config_3 
When Interface::handle_config_3 (third step of applying a new configuration to
interfaces) tried to detach the interface from the current IP config because
the old and new IP config differed, it did so using the new domain. The former
steps of the reconfiguration already installed the new domain reference at the
interface. Therefore, also the DHCP server of the new domain was used. This,
however caused uncaught exceptions because detaching from an IP config
includes dissolving all DHCP allocations. This dissolving of DHCP allocations
now operated on a DHCP server (the one of the new domain) that wasn't related
to the allocations and, in the worst case, caused an uncaught exception
because the IPs were out of its range.

That said, this commit ensures that detaching an interface from an IP config
is always done on the domain from which the IP config originated. Normally,
this is the domain the interface is attached to. But in the case of
Interface::handle_config_3, it is another - the former domain the interface
was attached to.

The commit also adapts the nic_router_dhcp_* tests in a way that they
reconfigure the router in a way that would trigger the uncaught exception
without the fix.

Fixes #4200
 2021-06-25 11:57:24 +02:00
Sebastian Sumpf
fd9bc43be1 ldso: Fix RISC-V support with 21.05 tool chain 
- force inlining of more functions
- force -O2 for RISC-V (even if -O0 is set globally)
- intialize cxx guard after self relocation

issue #4094
 2021-06-25 11:56:22 +02:00
Sebastian Sumpf
a6fe6c90d4 tool_chain: add ENABLE_FEATURE support 
default: "c c++ ada go gdb"

This allows to build a subset of the Genode tool chain

issue #4094
 2021-06-25 11:55:41 +02:00
Christian Helmuth
ece33d37f8 depot: update recipe hashes 2021-06-25 11:46:37 +02:00
Stefan Kalkowski
e7067050be base: extend cache maintainance functionality 
Introduce two new cache maintainance functions:
* cache_clean_invalidate_data
* cache_invalidate_data
used to flush or invalidate data-cache lines.

Both functions are typically empty, accept for the ARM architecture.
The commit provides implementations for the base-hw kernel, and Fiasco.OC.

Fixes #4207
 2021-06-25 11:43:41 +02:00
Stefan Kalkowski
ed0cc5330e tool/dde_linux: add list_dependencies utility 
Fix #4204
 2021-06-25 11:43:41 +02:00
Johannes Schlatow
b83c8f35c6 depot: ffi for x86_32 
genodelabs/genode#4201
 2021-06-25 11:43:41 +02:00
Sebastian Sumpf
a242bfce48 libports: ffi for x86_32 
- Required by glib

issue #4201
 2021-06-25 11:43:41 +02:00
Sebastian Sumpf
19a7997734 qemu-usb: consume device report during controller construction 
Initially parse the USB devices report after the signal handler is
registered, do not rely on a signal to be triggered (issue #1788).

Fixes #4206
 2021-06-25 11:43:41 +02:00
Sebastian Sumpf
a58473dece autopilot: add usb_hid_reconnect run script 
- Adjust script to run on Qemu x86 only

fixes #4205
 2021-06-25 11:43:41 +02:00
Christian Helmuth
cd25dc4e6a dde_bsd: prevent ‘nonnull’ argument warning 
The implementation conflicted with the implicit declaration of bzero:

.../repos/dde_bsd/src/lib/audio/mem.cc: In function ‘void bzero(void*, size_t)’:
.../repos/dde_bsd/src/lib/audio/mem.cc:377:2: warning: ‘nonnull’ argument ‘b’ compared to NULL [-Wnonnull-compare]
 2021-06-25 11:41:46 +02:00
Christian Helmuth
c585e008b1 acpica: prevent recursive 'memset()' calls with gcc 10 
Thanks to Peter for the patch.

Issue #4125
 2021-06-25 11:41:45 +02:00
Norman Feske
5b85bd9602 mk: make LD_SCRIPT_DYN customizable 
This way, a target.mk becomes able to override the default linker
script, e.g., for adding custom symbols.
 2021-06-25 11:41:45 +02:00
Martin Stein
0dc7084b0f wifi_drv: fix compile warnings with gcc 10 
Fixes #4202
 2021-06-25 11:41:45 +02:00
Martin Stein
16c4aacf34 vfs/dir_fs: allow opening empty dirs 
Adapts Dir_file_system::open_composite_dirs in a way that it returns "success"
when the leaf node of the path is an empty directory but "lookup failed", as
usual, if one of the other directories on the way to the leaf node is empty.

I couldn't find a technical reason why we used to return "lookup failed" when
only the leaf node was empty.

The commit also adds a test for en empty root directory and empty
sub-directories to the fs_query run script.

Fixes #4198
 2021-06-25 11:41:45 +02:00
Martin Stein
6bfdddd0b5 fs_query: gracefully deal with missing directories 
The fs_query component used to exit with an uncaught exception if a queried
directory didn't exist. Now, fs_query will catch this event and simply skip the
affected query, thereby indicating to the user the inexistence of the
queried directory.

Ref #4032
 2021-06-25 11:41:45 +02:00
Johannes Schlatow
026b117a63 separate sandbox api from os 
genodelabs/genode#4191
 2021-06-25 11:41:45 +02:00
Norman Feske
e5600fea06 dde_linux/extract_initcall_order: support v5.12 
This patch adjusts the 'extract_initcall_order' utility to the changed
initcall symbol syntax introduced in Linux by the following commit:

   https://github.com/torvalds/linux/commit/a8cccdd9

It uses an additional sed step to strip away the kmod prefix,
__COUNTER__, and __LINE__ information so that the symbol names match
those of earlier kernel versions.

Issue #4188
 2021-06-25 11:41:45 +02:00
Norman Feske
f541668604 dde_linux/create_dummies: show objs for variables 
Issue #4188
 2021-06-25 11:41:45 +02:00
Norman Feske
bf7500ad7b dde_linux/create_dummies: out-of-tree build dir 
This patch enables the tool/dde_linux/create_dummies tool to operate on
a out-of-tree build directory. The Linux source tree is determined by
examining the build directory's Makefile.

Issue #4188
 2021-06-25 11:41:45 +02:00
Norman Feske
29b7c5a202 dde_linux/create_dummies: show symbol details 
When combining the 'show' command with a defined LINUX_KERNEL_DIR,
each symbol gets annotated with the object file where it is defined.

Issue #4188
 2021-06-25 11:41:45 +02:00
Johannes Schlatow
7346defc26 usb_modem_drv: remove nic_server mode 
genodelabs/genode#4201
 2021-06-25 11:41:45 +02:00
Johannes Schlatow
dff1df0b49 usb_modem: do not fail on netif_stop_queue 
genodelabs/genode#4201
 2021-06-25 11:41:45 +02:00
Johannes Schlatow
2c87c68a5d depot: add ffi api and src archives 
Fixes genodelabs/genode#4201
 2021-06-25 11:41:45 +02:00
Johannes Schlatow
52a4293bbc depot: add usb_modem_drv archives 
genodelabs/genode#4201
 2021-06-25 11:41:44 +02:00
Sebastian Sumpf
ff57bf617b usb_webcam: webcam run script 
Run script for quickly testing webcams in Qemu or on target.

issue #4196
 2021-06-25 11:41:44 +02:00
Sebastian Sumpf
818f1682ee qemu-usb: flush EP improve isochronous handling 
- Patch the XHCI model in order to handle frame wrapping correctly. For
  this adjust 'mfindex_kick' to the correct period (same, before, or after
  'mfindex').

- Flush EP when it is stopped, this causes all pending packets for the EP
  to be acked. Correct counting of packets in flight.

- Add BEI patch by Josef.

issue #4196
 2021-06-25 11:41:44 +02:00
Sebastian Sumpf
eabda8907f usb_host: add support to flush EP transfers 
This causes all pending transfers of a EP to be canceled and in turn to
be acked by the raw session.

issue #4196
 2021-06-25 11:41:44 +02:00
Alexander Boettcher
4aa99fd1a9 qemu-usb: reduce cpu load by webcam model 
if no new frame is available for capturing

Fixes #4078 #4196
 2021-06-25 11:41:44 +02:00
Sebastian Sumpf
ff452619e3 vbox5-nova-capture: Set camera to 640x480@30fps 
Default to 30fps

issue #4196
 2021-06-25 11:41:44 +02:00
Sebastian Sumpf
19a5fee70b depot_deploy: add <capture> session announcement 
Announce Capture session if node is present in runtime.

issue #4196
 2021-06-25 11:41:44 +02:00
Sebastian Sumpf
c66a196f76 usb_webcam: package recipes 
- API packages for: libusb, libuvc, and libyuv
- Source packages for: API packages + USB webcam app
- Meta package for USB webcam
- Raw package for USB webcam configuration

issue #4196
 2021-06-25 11:41:44 +02:00
Sebastian Sumpf
9165c7601d libyuv: make shared object 
- Adjust port to Genode expected layout
- Make shared library
- Adjust dependent infrastructure to change

issue #4196
 2021-06-25 11:41:44 +02:00
Martin Stein
ff128df131 dde_linux/fec: fix unsynchronized TX DMA write 
Unfortunately, our current implementation of 'wmb()' doesn't seem to do what we
want it to do. On base-hw + imx6q_sabrelite, the write of bdp->cbd_sc seems to
get re-ordered after the write to txq->bd.reg_desc_active in the transmission
path of the contrib code. Due to this, the transmission of the packet is only
triggered the next time a packet is sent. However, we only quick-fix it by
enforcing the execution of the write with a volatile global read as we will
soon update the FEC NIC port with a new DDE approach anyway.

Fixes #4010
 2021-06-25 11:41:44 +02:00
Christian Prochaska
747d01e854 qt5: adapt QtWebEngine contrib code for use of nss 
Fixes #4192
 2021-06-25 11:41:44 +02:00
Christian Helmuth
331844c979 vbox6: support capslock="rom" mode 
In ROM mode the global CapsLock state is controlled by the capslock ROM
by virtual KEY_CAPSLOCK events.

Guests are easily confused by spurious KEY_CAPSLOCK input events in
caps="rom" mode. These spurious events may reach the VMM if KEY_CAPSLOCK
is not pressed as first key in a combination and, therefore, is not
filtered as global key. We filter KEY_CAPSLOCK in ROM mode in the VMM
explicitly, but let it pass in non-ROM mode.

Per default RAW mode is used and CapsLock key events are sent unfiltered
to the guest.
 2021-06-25 11:41:44 +02:00
Christian Helmuth
434d007dc1 stdcxx: some missing symbols for basic_string 
Revealed with -O0 when linking test/lx_fs_notify/file_writer.

Issue #4070
 2021-06-25 11:41:44 +02:00
Pirmin Duss
7db6f457d4 lx_fs: enable watch 
Enable watching files via the inotify interface of the Linux Kernel.

Delivery of watches to components is staggered in order to prevent an
overflow of the ACK queue in cases when a lot of changes are made to the
file system from the Linux side.

Fixes #4070
 2021-06-25 11:41:44 +02:00
Christian Helmuth
37f1873f2e vbox5: drop KEY_CAPSLOCK events on capslock="rom" 
Guests are easily confused by spurious KEY_CAPSLOCK input events in
caps="rom" mode. These spurious events may reach the VMM if KEY_CAPSLOCK
is not pressed as first key in a combination and, therefore, is not
filtered as global key. Now, we filter KEY_CAPSLOCK in ROM mode in the
VMM explicitly, but let it pass in non-ROM mode.

Fixes #4087
 2021-06-25 11:41:43 +02:00
Christian Helmuth
002037ce15 vbox6: support MAC address overwrite 2021-06-25 11:41:43 +02:00
Christian Helmuth
2a1a47b598 vbox6: update to version 6.1.22 2021-06-25 11:41:43 +02:00
Alexander Boettcher
ab31de0f6a libc_integration: reduce rounds for x86_32 
autopilot mode. The test machine is less powerful as the x86_64 one.

Issue #3967
 2021-06-25 11:41:43 +02:00
Pirmin Duss
a37ff1d985 lx_fs: enable session quota upgrades 
Issue #4070
 2021-06-25 11:41:43 +02:00
Pirmin Duss
4053e1628b lx_fs: support for unlink 
This is based on the work of @ehmry and @sidhussmann.

Issue #4070
 2021-06-25 11:41:43 +02:00
Alexander Boettcher
27004e1fd5 usb: remove monolithic usb driver 
+ adjust README files

Fixes #4173
 2021-06-25 11:41:43 +02:00
Johannes Schlatow
b09e69a444 ports: change bashrc path 
genodelabs/genode#4163
 2021-06-04 11:29:04 +02:00
Johannes Schlatow
758c0a21cc depot: add jschlatow download and pubkey 
Fixes #4185
 2021-06-03 08:23:50 +02:00
Christian Prochaska
60eec251e0 qt5: increase stack size of QtWebEngine render thread 
Fixes #4168
 2021-06-02 10:36:01 +02:00
Stefan Kalkowski
336350fe60 sculpt: enable support for MNT Reform2 2021-06-01 15:53:48 +02:00
Stefan Kalkowski
8408bf6ac0 fec_nic_drv: remove nic_server mode 
Issue #4133
 2021-06-01 15:51:22 +02:00
Alexander Boettcher
73d87073af vbox5/6: avoid negative timeouts in xhci model 
Fixes #4184
 2021-05-31 15:36:53 +02:00
Sebastian Sumpf
cf2527269f qemu-usb: allocate host devices after webcam 
Because qemu-usb allocated host devices after 'USB_HOST_DEVICE' in the
object array and 'USB_WEBCAM' is loacated after 'USB_HOST_DEVICE' the
webcam model can overwrite an already allocated pass-through device. As
a solution add the 'USB_FIRST_FREE' to make it clear from where host
devices can be allocated. Also increase the number of supported host
devices to eight.

fixes #4182
 2021-05-31 15:19:46 +02:00
Christian Prochaska
520b69ef0d libports: update expat port 
Fixes #4183
 2021-05-31 15:19:44 +02:00
Christian Helmuth
c0150f97e5 version: 21.05 2021-05-31 13:24:05 +02:00
Norman Feske
5dc7d55cc0 News item for version 21.05 2021-05-31 13:24:04 +02:00
Norman Feske
fd2f137a9b Release notes for version 21.05 2021-05-31 12:41:38 +02:00
Christian Helmuth
1d52bd017d depot: update recipe hashes 2021-05-28 14:16:46 +02:00
Josef Söntgen
e06f3bba27 drivers_managed-imx8q_evk: fix hdmi device node 
By now the 'compatible' name is set via the 'type' attribute.

Fixes #4181.
 2021-05-28 14:16:46 +02:00
Stefan Kalkowski
1d12755401 depot: add i2c api package 
Fixes #4180
 2021-05-28 14:16:46 +02:00
Stefan Kalkowski
f2ac341003 depot: add architectural content to os api package 
Fixes #4179
 2021-05-28 14:16:45 +02:00
Christian Prochaska
f2ff1a6d52 dde_linux: save and restore ARM FPU state in setjmp.S 
Fixes #4161
 2021-05-28 14:16:45 +02:00
Johannes Schlatow
4383579db6 window_layouter: improve focus history 
If no window has ever been focused, next() always returns an invalid
window id. As a consequence, there is no way to cycle through the focus
history without an explicit focus event (e.g. mouse hover).

Instead, next() should return the first window from the focus history if the
currently focused window is not present.

Fixes genodelabs/genode#4164
 2021-05-28 14:16:45 +02:00
Christian Helmuth
f0b9549376 wifi_drv: optionally report MAC address 
Issue #4133
 2021-05-28 14:16:45 +02:00
Christian Helmuth
aab6f52325 wifi_drv: remove nic_server mode 
Issue #4133
 2021-05-28 14:16:45 +02:00
Christian Helmuth
6e85a73a28 ipxe_nic_drv: optionally report MAC address 
Issue #4133
 2021-05-28 14:16:45 +02:00
Christian Helmuth
4abc530974 ipxe_nic_drv: remove nic_server mode 
Issue #4133
 2021-05-28 14:16:45 +02:00
Christian Helmuth
583ba0e9db ipxe_nic_drv: eliminate static constructors 
Issue #4133
 2021-05-28 14:16:45 +02:00
Christian Helmuth
62f83b7198 linux_nic_drv: optionally report MAC address 
Issue #4133
 2021-05-28 14:16:45 +02:00
Christian Helmuth
464f0eaf8b linux_nic_drv: remove nic_server mode 
Issue #4133
 2021-05-28 14:16:45 +02:00
Josef Söntgen
0f72a342f3 wifi_drv: fix setting 'bssid' in config 
The wpa_supplicant refuses to set the BSSID in case it is quoted.
Removing the quotes allows for specifying the BSSID in the
configuration.

Fixes #4175.
 2021-05-28 14:16:45 +02:00
Alexander Boettcher
3dff399fba netperf_*_usb: fix cap quota for sel4+hw x86 
Fixes #4016
 2021-05-28 14:16:45 +02:00
Alexander Boettcher
0aa99648d7 run: use usb/hid driver in VM tests 
Issue #4173
 2021-05-28 14:16:45 +02:00
Alexander Boettcher
6b9fcc9449 drivers_interactive-rpi: use usb host/hid driver 
Issue #4173
 2021-05-28 14:16:45 +02:00
Alexander Boettcher
f9c7947c45 drivers_interactive-pc: use usb host/hid driver 
Issue #4173
 2021-05-28 14:16:45 +02:00
Stefan Kalkowski
28189ba77a tool: simplify initcall order extraction 
The new tool `extract_initcall_order` generates a header file out of a
Linux kernel build, which provides a sorted array with the initcall function
names of the kernel build. The order states the order in which the initcall
have to be called. It gets extrated out of the Linux kernel System.map.

Fixes #4172
 2021-05-28 14:16:45 +02:00
Alexander Boettcher
fc5903c917 themed_decorator: upgrade cap and memory on demand 
Issue #4150
 2021-05-28 14:16:45 +02:00
Stefan Kalkowski
7f6f710bd2 imx8mq_platform_drv: introduce reset domains 
A reset domain can consist of one or several reset-pins
denoted by name that are assigned to a device.
When the device gets acquired via the Platform RPC API,
the pins are de-asserted, and asserted again when the
device gets released.

A configuration looks like the following:

  <device name="mipi_dsi>
    <reset-domain name="mipi_dsi_pclk"/>
    ...
  </device>

Fixes #4171
 2021-05-28 14:16:45 +02:00
Stefan Kalkowski
ed7d6c74f4 i2c: extend API to support transactions 
Introduces the notion of a transaction that consists of one or more
messages. Whereby a message has a read or write direction and consists
of one or more bytes.

Issue #4170
Fixes #4169
 2021-05-28 14:16:45 +02:00
Stefan Kalkowski
9f099bd61c base-hw: use generalized util/array.h 
Issue #4170
 2021-05-28 14:16:45 +02:00
Stefan Kalkowski
6780cf0790 base: introduce array utility 
Fixes #4170
 2021-05-28 14:16:45 +02:00
Christian Prochaska
7b197d54ed tool chain: update gdb to version 10.2 
Issue #4094
 2021-05-28 14:16:44 +02:00
Alexander Boettcher
933de21339 base: add platform_info to microkernel platforms 
Add to older platforms also the info ROM to avoid confusion/red messages.

Issue #4016
 2021-05-28 14:16:44 +02:00
Alexander Boettcher
509e5aa776 platform(x86): report features via platform_info 
Report via platform_info the capabilities of the kernel, e.g. ACPI and MSI.

With the commit the try-catch pattern on IRQ session creation by the platform
driver is avoided.

Issue #4016
 2021-05-28 14:16:44 +02:00
Christian Prochaska
e1abd2db4e base-linux: update arm32 seccomp policy 
Issue #4136
Issue #3466
 2021-05-28 14:16:44 +02:00
Christian Helmuth
faf491ce92 smartcard: replace usb_drv by usb_host_drv 2021-05-28 14:16:44 +02:00
Alexander Boettcher
98f524bb41 qemu-usb/webcam: signal capturing off earlier 
based on comment in Linux sources. Reduce fallback off-capturing detection
to 1s.

Issue #4078
 2021-05-28 14:16:44 +02:00
Sebastian Sumpf
7fcf9053b9 usb_hid: perform device destruction on unplug signal only 
- Do not perform desctruction on report updatea in EP because
  'unregister_device' may block on Led state 'update' (synchronous
  control message) leading to the driver being stuck because no more
  signals are received
- Check if device is present in 'submit_urb' calls

fixes #4166
 2021-05-28 14:16:44 +02:00
Sebastian Sumpf
6910b880e7 usb_host: make device creation/destruction robust 
- Signal device ready depending on state (ready or not) immediately or
  when "actconfig" is set
- Report new devices when ready
- Drain packet stream in case there is no device present (needed for
  synchronous operations at client side)
- Do not use 'session_device' on device destruction, check pointer
  directly instead

issue #4149
 2021-05-28 14:16:44 +02:00
Alexander Boettcher
708b7f4619 cpu_balancer.run: adjust quota for seL4 2021-05-28 14:15:28 +02:00
Alexander Boettcher
921a99bb9b tool/run: option to copy tftp served files (x86) 
Fixes #4153
 2021-05-28 14:15:28 +02:00
Norman Feske
a13dee8d19 tool/dts: for extracting content from dts files 
The new tool eases the inspection and pruning of device-tree source
files.

Fixes #4165
 2021-05-28 14:15:28 +02:00
Martin Stein
0069660958 rump: fix compiler warning 
Fixes two warnings of this type:

! src/lib/vfs/rump/vfs_rump.cc:214:52: warning: narrowing conversion of
! ‘(s.stat::st_mode & 64)’ from ‘__mode_t’ {aka ‘unsigned int’} to ‘bool’
! [-Wnarrowing]
!
!  214 |                           .executable = (s.st_mode & S_IXUSR) };
!      |                                         ~~~~~~~~~~~^~~~~~~~~~

Fixes #4160
 2021-05-28 14:15:28 +02:00
Johannes Schlatow
7b09675236 depot: add less recipe 
genodelabs/genode#4162
 2021-05-28 14:15:28 +02:00
Johannes Schlatow
4bed825956 depot: add diffutils recipe 
genodelabs/genode#4162
 2021-05-28 14:15:28 +02:00
Sebastian Sumpf
5135ff2dc2 usb_webcam: An app using libuvc for USB webcams 
issue #4158
 2021-05-28 14:15:28 +02:00
Sebastian Sumpf
4a56171a77 libports: port of libuvc 
by Josef Soentgen

issue #4158
 2021-05-28 14:15:28 +02:00
Sebastian Sumpf
ae5b4c9624 libusb: Make USB library more robust 
- Add reference count for open/close
- Add cancel transfer support
- Handle packet stream correctly

issue #4158
 2021-05-28 14:15:28 +02:00
Josef Söntgen
46c846ef91 libports: add iso handling to libusb 
issue #4158
 2021-05-28 14:15:28 +02:00
Sebastian Sumpf
c8c589d91a libyuv: enable MJPEG support 
issue #4158
 2021-05-28 14:15:28 +02:00
Josef Söntgen
5254930930 libports: update jpeg library to v9d 
issue #4158
 2021-05-28 14:15:28 +02:00
Martin Stein
70797fe879 fs_tool: add <new-file> operation 
Adds the <new-file> operation to the fs_tool. When configured, the
<new-file path="...">...</new-file> tag will cause creation or overwriting of
the file given through the 'path' attribute. The file will contain the text
content of the tag.

Ref #4032
 2021-05-28 14:15:28 +02:00
Norman Feske
100583e262 os/vfs.h: New_file utility 
This patch moves the utility from the app/text_area to os/vfs.h to make
it easier to use by other components. By hosting the 'New_file' as a
friend alongside the 'Directory', we can now pass a 'Directory' as
constructor argument, which is consistent with other utilities such as
'File_content'.

As a further improvement, the new version supports the implicit creation
of the directory hierarchy leading to the new file.

Issue #4032
 2021-05-28 14:15:27 +02:00
Norman Feske
57fbd2b658 os/vfs.h: Directory::create_sub_directory 
This method can be used to create a directory hierarchy within a
'Directory'.

Issue #4032
 2021-05-28 14:15:27 +02:00
Alexander Boettcher
9549eeeca4 ssh_terminal: avoid deadlock during bind callback 
Issue #4095
 2021-05-28 14:15:27 +02:00
Christian Helmuth
9166a75f2c foc: map uncached normal memory bufferable 
Mapping normal memory bufferable restores support for unaligned reads on
DMA memory and prevents the following errors on imx6q_sabrelite.

  KERNEL0: alignment error at 18003061 (PC: 0102e3f8, SP: 401ffb18, FSR: 90000001, PSR: 20000110)

Issue #4094
Issue #4157
 2021-05-28 14:15:27 +02:00
Martin Stein
dd587c6712 fs_query: support querying file sizes 
By adding an attribute 'size="yes"' to a query, one instructs fs_query to
report also the size of each queried file as attribute 'size' of the
corresponding 'file' node.

Ref #4032
 2021-05-28 14:15:27 +02:00
Martin Stein
3ed26e7bb2 fs_query: watch only readable files 
The fs_query component used to try watching all files it found resulting in
errors on files that are not watchable. For some files, however, the watch-
feature doesn't make sense as they are not readable (no content, no size).
Now, fs_query will check first whether a file is readable and skip watching
if it isn't.

Ref #4032
 2021-05-28 14:15:27 +02:00
Martin Stein
6e900f147c fs_query: read content only from readable files 
When configuring fs_query to print the content of files it used to try so for
all files it found resulting in errors on files that are not readable. Now,
fs_query will check first whether a file is readable and skip printing the
content of those that are not.

Ref #4032
 2021-05-28 14:15:27 +02:00
Tomasz Gajewski
0507d3f44b ssh_terminal: fixed managing ssh file descriptors 
Managing ssh event file descriptors was performed from two different
threads which could cause reallocation of structure used in other thread
in a call to 'poll' function.

Splitted initialization to parts and moved ssh event part into ssh loop.

Issue #4095
 2021-05-28 14:15:27 +02:00
Tomasz Gajewski
0b641ba581 ssh_terminal: fixed race condition during startup 
Moved creating ssh loop thread after initialization of wake up server
file descriptors to make sure that they will be properly handled even in
first loop run.

Issue #4095
 2021-05-28 14:15:27 +02:00
Martin Stein
b3a229eebb vfs_block: write less to the log 
Remove calls to the Genode log functions that were done unconditionally or
according to a static verbosity flag.

Ref #4032
 2021-05-28 14:15:27 +02:00
Martin Stein
fd0a4e78c8 rump: write less to the log 
Remove calls to the Genode log functions that were done unconditionally or
according to a static verbosity flag.

Ref #4032
 2021-05-28 14:15:27 +02:00
Stefan Kalkowski
74d826d1ad tool: simplify Linux dummy function generation 
The new tool `create_dummies` easily lists missing references,
while porting subsystems of the Linux kernel to Genode, e.g., device drivers.
Moreover it can automatically generate dummy implementations for functions
and global variables.

Fixes #4156
 2021-05-28 14:15:27 +02:00
Alexander Boettcher
b6b9801c20 seoul: fix ahci model busy bit handling 
A warning, discovered during GCC 10.3 update, got fixed wrongly.

Issue #4097
 2021-05-28 14:15:27 +02:00
Stefan Kalkowski
b22d9385f1 imx8mq_platform_drv: enable root clock div setting 
Fixes #4155
 2021-05-28 14:15:27 +02:00
Stefan Kalkowski
516601b7ba os: add Platform::Device::sigh_omit_initial_signal 
Add possibility to register an interrupt handler without triggering
an initial spurious interrupt for ported device driver code.

Fixes #4154
 2021-05-28 14:15:27 +02:00
Alexander Boettcher
67a8f29697 rump_ext2.run: increase cap quota for sel4 2021-05-28 14:15:27 +02:00
Josef Söntgen
3e284558a1 drm: account for guard page 
This makes the rendering issue with mesa gears disappear.

Issue #4148
 2021-05-28 14:15:27 +02:00
Alexander Boettcher
04d8e03ecb mesa: use 32bit RGB in egl platform lib 
Fixes distortion in mesa gears i965

Issue #4148
 2021-05-28 14:15:27 +02:00
Alexander Boettcher
cfe29b0e52 libc_integration.run: specify random source 
After update of stdcxx, either hardware (CPU) random sources are taken
or, if not available/insufficient, /dev/urandom is used.

Issue #3967
Issue #4094
 2021-05-28 14:15:27 +02:00
Alexander Boettcher
d4b9be8d44 stdcxx: support /dev/urandom 
when cpu does not provide sufficient hardware support. (happens on our
nightly test machine as also build machine).

Issue #4094
 2021-05-28 14:15:27 +02:00
Johannes Schlatow
db97af8dec vfs: fix read of large files in fs_file_systems 
For fs_file_systems, reads are limited to the size of the packets from the
File_system session. Hence, we cannot read the large files in one go.

This fix is particularly helpful for fonts_fs, as it enables including font
files from a File_system.

genodelabs/genode#4135
 2021-05-28 14:15:27 +02:00
Johannes Schlatow
4daf19ec7e vfs_ttf: update glyphs when font file changes 
genodelabs/genode#4135
 2021-05-28 14:15:27 +02:00
Christian Prochaska
18a43c1afd base-foc: fix assignment in 'Ipc_pager::acknowledge_exception()' 
Issue #4094
 2021-05-28 14:15:26 +02:00
Christian Helmuth
813f4d976b lxip: allocate skb with properly aligned IP header 
Comment in Linux sources:

  Since an ethernet header is 14 bytes network drivers often end up with
  the IP header at an unaligned offset. The IP header can be aligned by
  shifting the start of the packet by 2 bytes. Drivers should do this
  with:

  skb_reserve(skb, NET_IP_ALIGN);

This is ensured when using netdev_alloc_skb_ip_align().

Issue #4094
 2021-05-28 14:15:26 +02:00
Sebastian Sumpf
a24224ffc3 usb_host: improve isochronous performance on Intel 
This patch takes advantage of block transfer interrupts on Intel XHCI
controllers which is used during isochronous transfers. Because of a bug
in hardware (see usb_host_isoc_bei.patch header), this feature has been
disabled for Intel leading to up to 8000 interrupts/s for isochronous
transfer causing severe CPU consumption on Genode. With this commit we
lower host driver consumption to normal levels.

issue #4149
 2021-05-28 14:15:26 +02:00
Sebastian Sumpf
9543161827 usb_host: Move task to device in USB session 
A Lx::Task is now associated to a USB device not the session any more.
This implies that a task lives as long as the device making it possible
to gracefully handle outstanding requests (i.e., synchronous) in case
the session has been closed.

issue #4149
 2021-05-28 14:15:26 +02:00
Martin Stein
a0517686ca ports/libsparkcrypto: update for GCC 10 
The old port version contained '*.ali' files that were build with an older GCC
which led to problems when compiling packages that use the port with the new
GCC 10. The '*.ali' files of the new port version were generated with GCC 10.

Fixes #4145
 2021-05-28 14:15:26 +02:00
Sebastian Sumpf
9209dfc9d5 ldso: output loaded objects of 'Shared_object' interface 
When loading shared libraries via the 'Shared_object' interface display
all additionaly loaded libraries in case 'ld_verbose' is configured. Up
until now, only the loaded library was displayed. In order to determine
if a dependend library had arlready been loaded prior to loading the
'Shared_object' the reference counter is used.

fixes #4147
 2021-05-28 14:15:26 +02:00
Alexander Boettcher
f8d0552d52 libc_integration: fix out-of-bound access 
which leads to sporadic pagefaults/segmentation faults.

Fixes #3967
 2021-05-28 14:15:26 +02:00
Martin Stein
139a2cfae9 tool: use local LC_ALL=C instead of LC_COLLATE=C 
We used to export 'LC_COLLATE=C' for the environment of Genode tools. This was
meant to ensure that sorting is always done C-style and not dependent on the
users locale settings. This is required, for instance, to ensure that the same
archive always yields the same hash.

However, 'export LC_COLLATE=C' is not sufficient. It can be outruled by an
'LC_ALL' setting in the users environment. The manual of 'sort' recommends to
set 'LC_ALL=C' locally if you want reliable results and this is what this
commit does. Furthermore it removes the former 'export LC_COLLATE=C'
directives.

Note that I couldn't find a way to set 'LC_ALL' local to 'exec ... sort' in
TCL. This is why I set it global instead using TCLs 'env' array.

Note that the Make directive '$(sort ...)' and the TCL directive 'lsort',
unlike the Shell directive 'sort', are not affected by the users locale
settings.

Fixes #4144
 2021-05-28 14:15:26 +02:00
Alexander Boettcher
1c20ed12c1 qemu: sanity check misconfigured memory for okl4 
Issue #4095
 2021-05-28 14:15:26 +02:00
Alexander Boettcher
d516544a1f ssh_exec_channel.run: use default Qemu memory size 
The default size is on most kernels 512M. On OKL4 we have to use 800M,
because of the statically configured memory ranges in the OKL4 kernel.

By avoiding a specific amount of memory, the default Qemu memory sizes are
used.

Issue #4095
 2021-05-28 14:15:26 +02:00
Martin Stein
f8b2fd522b ports/ada-runtime: update to support gcc 10 
Fixes #4128
 2021-05-28 14:15:26 +02:00
Christian Prochaska
471d34a367 Remove mpc and mpfr ports 
These ports are not needed anymore with the 21.05 tool chain update.

Issue #4094
 2021-05-28 14:15:26 +02:00
Christian Prochaska
d5e4ffd191 sanitizer: update to version 10.3.0 
Issue #4094
 2021-05-28 14:15:26 +02:00
Christian Prochaska
4a4b754bf2 gcov: update to version 10.3.0 
Issue #4094
 2021-05-28 14:15:26 +02:00
Christian Prochaska
a0f5d34e25 stdcxx: update to version 10.3.0 
Issue #4094
 2021-05-28 14:15:26 +02:00
Christian Prochaska
9a3d5dcf21 tool_chain: update ali2dep for gcc 10.3.0 
Issue #4094
 2021-05-28 14:15:26 +02:00
Christian Prochaska
9c05cda6e7 tool_chain: update gcc to version 10.3.0 
Issue #4094
 2021-05-28 14:15:26 +02:00
Sebastian Sumpf
6639261126 binutils: augment equally-named sections again 
Revert GNU ld to the old behavior where sections with the same name in multiple
ld scripts are merged. Binutils 2.36.1 creates two sections with the same name.

Fixes #4126
 2021-05-10 11:52:27 +02:00
Christian Prochaska
fc902d9e66 tool_chain: update binutils to version 2.36.1 
Issue #4094
 2021-05-10 11:19:13 +02:00
Christian Prochaska
6a11b78cdf tool_chain: use gcc-provided dependency libraries 
Download gmp, mpc and mpfr with the download script provided by the
gcc source tree and let the gcc build system handle the build of these
libraries with the correct compile options. This fixes build issues on
armhf Linux and removes the need to maintain mpc and mpfr ports in
the Genode tree.

Issue #4094
 2021-05-10 11:19:11 +02:00
Christian Prochaska
7fd598f0c1 tool_chain: build 'gprbuild' from source 
The 'gprbuild' version which is provided by Ubuntu 18.04 has a bug
which prevents it from building 'ali2dep' with gcc/gnat 10. With
this commit, a newer 'gprbuild' version is built from source.

Issue #4094
 2021-05-10 11:19:09 +02:00
Christian Helmuth
777923f9bd depot: update recipe hashes 2021-05-10 11:18:12 +02:00
Martin Stein
d2bf565503 nic/zynq: fix packed conversion error with gcc 10 
Fixes #4142
 2021-05-10 11:17:21 +02:00
Alexander Boettcher
384a8da50b ssh_terminal: use pthread_mutex 
to avoid sporadic deadlocks between EP thread and the server loop pthread.

Issue #4095
 2021-05-10 11:17:20 +02:00
Alexander Boettcher
b6bdd91cfa nova: avoid unaligned warnings by gcc10 
Issue #4126
 2021-05-10 11:17:20 +02:00
Christian Helmuth
a5385cebf4 block_cache: explicitly instantiate template function 
The explicit instantiation is required because the function is defined
in the compilation unit only but referenced externally.

Fixes #4108
 2021-05-10 11:17:20 +02:00
Josef Söntgen
c5c3d7ca98 gpt_write: address unaligned access for name 
Turn member into byte array to prevent unaligned memory access.

Fixes #4141.
 2021-05-10 11:17:20 +02:00
Norman Feske
d19b751632 nano3d: remove surface-clear optimization 
This patch fixes a GCC-10 compile error. Even though the optimization
was quite effective - I measured a speedup of factor 2 - it is not all
that important for the overall application performance. In the nano3d
case, we are talking about 1 vs. 2 percent of CPU time.

Fixes #4140
 2021-05-10 11:17:20 +02:00
Christian Prochaska
ff160decec gdb: fix build errors with gcc 10 
Fixes #4139
 2021-05-10 11:17:20 +02:00
Norman Feske
a4115cfea9 base-linux: discharge timestamp on AARCH64 
The kernel-agnostic 'Trace::timestamp' function for arm_64 executes the
'mrs %0, pmccntr_el0' instruction, which is not permitted for user-level
programs on Linux. This patch shadows the generic timestamp.h header
with dummy that returns zero. This return value prompts the timeout
framework to disable the interpolation of time based on timestamps. This
avoid the illegal-instruction abort but comes with two llimitations:

First, time measurements are effectlively limited to a granulariry of 1
millisecond (deliberately constrained by the timer driver).

The quirk is applied when using the base-linux API. Should a generic
application (that uses the base API only) call 'Trace::timestamp'
directly, the illegal instruction is executed.

Issue #4136
 2021-05-10 11:17:20 +02:00
Norman Feske
2f9d430c00 base-linux: 64-bit ARM support 
This patch adds support for running Genode/Linux on the AARCH64
architecture.

- The kernel-agnostic startup code (crt0) had to be extended to
  capture the initial stack pointer, which the Linux kernel uses
  to pass the process environment. This is in line with the
  existing startup code for x86_32 and x86_64.

- The link order of the host libraries linked to lx_hybrid
  programs had to be adjusted such that libgcc appears at last
  because the other libraries depend on symbols provided by
  libgcc.

- When using AARCH64 Linux as host, one can execute run scripts
  via 'make run/<script> KERNEL=linux BOARD=linux' now.

Issue #4136
 2021-05-10 11:17:20 +02:00
Christian Helmuth
718f44ae5b Check max_len before dereferencing pointer in Cstring constructor 
Fixes #4112
 2021-05-10 11:17:20 +02:00
Martin Stein
66feb939e6 cbe_tester: compile errors/warnings with gcc 10 
Fixes #4113
 2021-05-10 11:17:20 +02:00
Christian Helmuth
1baf844e20 vfs_stress: fix handling of 'char' in error message 
Issue #4112
 2021-05-10 11:17:20 +02:00
Martin Stein
5fa2efa745 net/dhcp: print readable message types 
When a DHCP packet is printed out, it first tries to determine the most
specific message type from the DHCP options and print its human-readable name
right after the protocol name. If finding the message type fails, the less
specific opcode is printed instead, but also in a human-readable way.

Fixes #4131
 2021-05-10 11:17:20 +02:00
Norman Feske
ad847d0543 sculpt: fix glitch in fs-browser back arrow 
Fixes #4124
 2021-05-10 11:17:20 +02:00
Norman Feske
0b36d81c0c base-linux: separate x86-specific code 
By moving core's build-description to a library - a pattern already
employed for the other base platforms - we become able to cleanly split
x86-specific code (I/O-port access) from generic code. This is a
prerequisite for enabling non-x86 architectures such as AARCH64.

Issue #4136
 2021-05-10 11:17:20 +02:00
Jean-Adrien DOMAGE
a4727c90a8 lx_fs: add os/run/lx_fs_import.run run script 
This new runscript demonstrates how a file can be overwritten by vfs
import with the flag overwrite set to false. This is due to a bug
in lx_fs that is not propagating an ernno EEXIST syscall error.

Issue genodelabs#4104
 2021-05-10 11:16:24 +02:00
Jean-Adrien DOMAGE
e65b7f3b82 lx_fs: fix errno EEXIST not propagated 
This commit introduces a fix for lx_fs to propagate errno EEXIST error.
This prevents vfs <import overwrite="false"> to overwrite an imported
file, if it already exists, without the overwrite flag set to true.

Issue genodelabs#4104
 2021-05-10 11:16:24 +02:00
Josef Söntgen
0f0edc0134 usb_host: implement sg_pcopy_* as TRACE dummies 
The former implementation called 'TRACE_AND_STOP' which led to the
'usb_host_drv' to stop whenever such a function was called. Rather
than stopping return '0' and let the contrib code deal with this
situation.

Issue #4116.
 2021-05-10 11:16:24 +02:00
Josef Söntgen
d0f084d449 usb_host: backport xhci bounce-buffer fix 
This commit backports the Linux upstream commit (d4a6106) [*]
that introduces a check to prevent the sglist from being used
unconditionally.

[*] 'xhci: fix bounce buffer usage for non-sg list case'

Fixes #4116.
 2021-05-10 11:16:24 +02:00
Christian Prochaska
c7b2314d23 foc: fix errors with gcc 10 and binutils 2.36 
Fixes #4118
 2021-05-10 11:16:24 +02:00
Christian Helmuth
eb94f03416 microcode.run: do not twiddle with MAKEFLAGS 
Unconditionally setting MAKEFLAGS to just 's' is not nice because other
flags cannot be passed through the run script. What the script desires
is to disable the automatically enabled '--print-directory' for sub-make
instances when calling tool/port/current by declaring
'--no-print-directory'.
 2021-05-05 11:55:27 +02:00
Norman Feske
516a9a6925 depot: update recipe hashes 2021-05-05 11:35:31 +02:00
Norman Feske
fdb1a4dd88 base-linux: avoid legacy syscalls 
Until now, Genode's Linux system call bindings were based on original
Unix system calls that were later superseded by more flexibile variants.
E.g., 'openat' is a modern version of 'open'. Even though Linux upholds
the compatiblity with the original versions for existing architectures
like x86, the legacy syscalls are absent for the recently added AARCH64
architecture. A good overview of the system calls accross the prominent
architectures can be found at

https://chromium.googlesource.com/chromiumos/docs/+/master/constants/syscalls.md

This patch updates Genode's syscall bindings to avoid legacy versions,
thereby easing the support for AARCH64. The used "modern" versions
were introduced back in Linux version 2 days. So we are fine to rely
on them.

The patch slightly changes the signature for lx_stat because this system
call is merely used to check for the existance of a file and its size.
The new name 'lx_stat_size' draws a closer connection to its use case.
That said, the stat syscall has not been updated to the modern statx
since statx is still a fairly recent addition.

Issue #4136
 2021-05-05 11:35:31 +02:00
Norman Feske
d477062c56 base-linux: simplify clone syscall binding 
This patch simplifies the use of the clone system call for creating
processes and threads. Until now, the binding used an opaque pointer
argument to pass context information to the newly created process or
thread. However, upon close inspection, this is not a strict
requirement.

A newly created thread accesses its contextual information by
using its stack pointer as key. The pointer argument is not used.

The creation of processes is strictly serialized because the
intermediate stack used in-between clone and execve is a global
variable. Since we rely on the serialization anyway, we can pass the
context information of a new process via a global variable as well.

This change simplifies the syscall binding for the upcoming AARCH64
support, which would otherwise require us to deal with the notion
of TLS on Linux.

Issue #4136
 2021-05-05 11:35:31 +02:00
Norman Feske
811009d18b News item about Sculpt 21.03b update 2021-05-05 11:35:31 +02:00
Christian Prochaska
00d13cf304 qt5: fix build errors with gcc 10 
Fixes #4132
 2021-05-05 11:35:31 +02:00
Martin Stein
49184fb938 net: circumvent stringop-overflow error 
With the update to GCC 10, the compiler stopped with an error when compiling
places where a MAC address is copied from outside into a packed object using
the Net::Netaddress::copy method (e.g. in
Net::Arp_packet::dst_mac(Mac_address)):

! error: writing 6 bytes into a region of size 4 [-Werror=stringop-overflow=]

While trying to find a clean solution for this error, I found posts on
gcc.gnu.org and github that stated that the size calculations that cause these
errors are incorrect. Indeed, I could verify that the actual size of the two
regions was static and exactly the same in places were the error occured.

Furthermore, I couldn't find a way of making it more clear to the compiler
that the sizes are the same. By accident, we found that using the address of
the first element of the array that forms the second region instead of the
array address itself, somehow circumvents the error.

Fixes #4109
 2021-05-05 11:35:31 +02:00
Martin Stein
52e8c95321 net: fix packed-conversion compiler warning 
With the update to GCC 10 the compiler used to warn when using the internet
checksum functions on packet classes (like in
Net::Ipv4_packet::update_checksum):

warning: converting a packed ‘Net::[PACKET_CLASS]’ pointer
         (alignment 1) to a ‘const uint16_t’ {aka ‘const short
         unsigned int’} pointer (alignment 2) may result in an
         unaligned pointer value

Apparently, the 'packed' attribute normally used on packet classes sets the
alignment of the packet class to 1. However, for the purpose of the
internet-checksum functions, we can assume that the packet data has no
alignment. This is expressed by casting the packet-object pointer to a pointer
of the new packed helper struct 'Packed_uint16' that contains only a single
uint16_t member before handing it over to the checksum function (instead of
casting it to a uint16_t pointer).

Ref #4109
 2021-05-05 11:35:31 +02:00
Christian Prochaska
9b854e1496 Prevent recursive 'memset()' calls with gcc 10 
Fixes #4125
 2021-05-05 11:35:31 +02:00
Christian Prochaska
d5e7870532 virtualbox5: fix build errors with gcc 10 
Fixes #4106
 2021-05-05 11:35:31 +02:00
Christian Prochaska
bc8dbc6b1a bash: fix build error with gcc 10 
Fixes #4107
 2021-05-05 11:35:31 +02:00
Christian Prochaska
8f23e377d9 lxip: fix build error with gcc 10 
Fixes #4102
 2021-05-05 11:35:31 +02:00
Christian Prochaska
aebece5110 readline: fix build error with gcc 10 
Fixes #4103
 2021-05-05 11:35:31 +02:00
Christian Prochaska
82604f2c2b lx_fs: fix build errors with gcc 10 
Fixes #4123
 2021-05-05 11:35:31 +02:00
Norman Feske
4563baae77 app/text_area: fix build error with gcc 10 
Fixes #4098
 2021-05-05 11:35:30 +02:00
Christian Prochaska
c4e3d3dbc4 hw: fix build error with gcc 10 
Fixes #4115
 2021-05-05 11:35:30 +02:00
Alexander Boettcher
eb53f8c113 seoul: adjust to GCC 10.03 
Fixes #4097
 2021-05-05 11:35:30 +02:00
Christian Prochaska
07649b667b sel4: fix build errors with gcc 10 
Fixes #4121
 2021-05-05 11:35:30 +02:00
Christian Prochaska
0f679bb35e pistachio: fix build error with gcc 10 
Fixes #4119
 2021-05-05 11:35:30 +02:00
Christian Prochaska
d6e9d74038 nitpicker: fix build error with gcc 10 
Fixes #4110
 2021-05-05 11:35:30 +02:00
Christian Prochaska
405e6744fb nova: update kernel for gcc 10 
Fixes #4114
 2021-05-05 11:35:30 +02:00
Christian Prochaska
fad85c3fd5 virtualbox6: fix build errors with gcc 10 
Fixes #4105
 2021-05-05 11:35:30 +02:00
Christian Prochaska
49392dfa44 dde_ipxe: fix build error with gcc 10 
Fixes #4101
 2021-05-05 11:35:30 +02:00
Christian Prochaska
e627f8320f verify: fix build error with gcc 10 
Fixes #4099
 2021-05-05 11:35:30 +02:00
Christian Prochaska
8196e229b0 netperf: fix build error with gcc 10 
Fixes #4096
 2021-05-05 11:35:30 +02:00
Christian Prochaska
0a8394c891 depot_autopilot: store signal context capability as copy 
Fixes #4129
 2021-05-05 11:35:30 +02:00
Christian Prochaska
66e8f8d764 base: increase bss alignment to 8 bytes 
Fixes #4127
 2021-05-05 11:35:30 +02:00
Christian Prochaska
84e4cbb54c cxx: add 'strchr()' 
Fixes #4122
 2021-05-05 11:35:30 +02:00
Tomasz Gajewski
23b21812dd Fixes in gdb_monitor_interactive/target_config.run 
Fixes #4134
 2021-05-05 11:35:30 +02:00
Norman Feske
d66e55ec37 aes_cbc_4k.run: increase timeout 
needed for foc/qemu/x86_32
 2021-05-05 11:35:29 +02:00
Martin Stein
5dbc9ef244 nic_router: more descriptive port allocator code 
This renames some members and local variables in the port allocator in order to
make the code more descriptive.

Fixes #4086
 2021-05-05 11:35:29 +02:00
Sid Hussmann
4e822436fc nic_router: use increasing src port for new nat 
The NAT feature of the NIC router used to prefer re-using source ports that
have been freed recently. From an external server's perspective, if a client
dies and restarts, chances are high that the new connect arrives with the same
source-IP/source-port as the old connection. The server has to forcefully reset
the connection. If that happens a lot, the server may even start to ignore
further connections from this IP/port combination for a while as a mitigation.

This patch adds a continuous counter feature that makes sure that every new
port allocation will increment and result in a port that hasn't been used for a
long time.

The NAT feature of the nic_router is now more in line with RFC 6056 chapter 4.

Ref #4086
 2021-05-05 11:35:29 +02:00
Alexander Boettcher
ce75b25fd4 ssh_exec_channel: adjust cap quota for seL4 
Issue #4095
 2021-05-05 11:35:29 +02:00
Josef Söntgen
118e8ee6e1 ssh_terminal: flush buffer on Terminal cleanup 
Issue #4095.
 2021-05-05 11:35:29 +02:00
Alexander Boettcher
f236e99b5c ssh_terminal: avoid deadlock of EP and pthread.0 
pthread.0 acquires a write buffer mutex and calls potentially
blocking fs operations. The EP thread handles session requests and tries to
acquire the same write buffer lock. IO progress events for pthread.0 are
handled by the EP thread, which however is blocking on the write buffer mutex.

The commit uses two write buffers, one which is filled by the EP and a second
which is used by pthread.0. The two buffers are swapped protected by a mutex
without invoking blocking fs operations.

Issue #4095
 2021-05-05 11:35:29 +02:00
Alexander Boettcher
86e09b60c4 exec_terminal: sandbox errors about unknown node 
The exec_terminal generates a config report which is used as input ROM for
sandbox/init, which does not support an "<empty>" node.

An empty config node is empty, without an node.

Issue #4095
 2021-05-05 11:35:29 +02:00
Johannes Schlatow
373b45a0f0 depot: fix content.mk of drivers_managed-pc 
Was still using the event_filter.config from drivers_interactive-pc
although a dedicated file is present in the raw archive.

The fix is just for consistency reasons, as sculpt manager is generating the
event_filter.config anyway.
 2021-05-05 11:35:29 +02:00
Johannes Schlatow
a2491c30b3 event_filter: add new test cases 
genodelabs/genode#4069
 2021-05-05 11:35:29 +02:00
Johannes Schlatow
d80b2a150a event_filter: add <log> node 
genodelabs/genode#4069
 2021-05-05 11:35:28 +02:00
Johannes Schlatow
c802de2cf9 event_filter: add <ignore-key> node 
Fixes genodelabs#4069
 2021-05-05 11:35:28 +02:00
Johannes Schlatow
e86387d557 event_filter: allow remapping of KEY_UNKNOWN 
KEY_UNKNOWN is a collective symbols for all unknown keycodes.
Remapping thus requires iterating through all corresponding codes
instead of only applying the policy to the first match.

Issue genodelabs#4069
 2021-05-05 11:35:28 +02:00
Norman Feske
f6aabfe233 Sculpt: prevent nitpicker's resize warnings 
This patch increases the RAM quota of the top-level nitpicker instance
to 12 MiB so that resize operations can be buffered for resolutions up
to 2560x1080. It mutes diagnostic messages of the form "Warning: Gui
(...) not enough RAM to preserve buffer content during resize" and
improves the window-resize experience.
 2021-05-05 11:35:28 +02:00
Norman Feske
8617e5cee0 Sculpt: un-inspect failed USB storage 
With this patch, the user is able to re-gain access to the inspect view
by toggling the inspect button of a file system unrelated to the failed
USB storage device.

Fixes #4090
 2021-05-05 11:35:28 +02:00
Johannes Schlatow
2db94b8438 platform_drv/x86: support to disable MSI-X 
+ disable for wifi driver MSI-X

Issue #4079
 2021-05-05 11:35:28 +02:00
Alexander Boettcher
7b9e7361ba vbox5: add capture pkg version 
Issue #4078
 2021-05-05 11:35:28 +02:00
Alexander Boettcher
910ea16405 qemu-usb: restrict build to x86 
Issue #4078
 2021-05-05 11:35:28 +02:00
Alexander Boettcher
58db8c647a qemu-usb: add usb webcam model 
Fixes #4078
 2021-05-05 11:35:28 +02:00
Alexander Boettcher
4826bd82fe qemu-usb: fix spelling 
Issue #4078
 2021-05-05 11:35:28 +02:00
Alexander Boettcher
7b90f8f857 libports: add libyuv support 
Issue #4078
 2021-05-05 11:35:28 +02:00
Johannes Schlatow
bf3ad3baff vbox5/6: pass extra mouse buttons to VM 
Fixes #4120
 2021-05-05 11:35:28 +02:00
Johannes Schlatow
8a4b52d9e3 vbox: only clear screen if virtual_fb_mode changed 
Fixes genodelabs/genode#4085
 2021-05-05 11:35:28 +02:00
Johannes Schlatow
a47fd36b9f vbox: fix high VM load after resize 
break circle of repeated guest notifications if the guest never
acknowledges a VMMDEV_EVENT_DISPLAY_CHANGE_REQUEST

genodelabs/genode#4084
 2021-05-05 11:35:28 +02:00
Christian Prochaska
a2e62db6ec qt5: fix libpng vulnerabilities 
Fixes #4089
 2021-05-05 11:35:28 +02:00
Alexander Boettcher
7503472ae6 stdcxx: add missing symbols for istream and thread 
Issue #3967

triggers using -O0 with test-libc_integration:

libc_integration/main.cc:146: undefined reference to `std::istream::operator>>(unsigned long&)'

contrib/stdcxx-80f380143250d4f951433876698b54fdac32b95f/include/stdcxx/std/thread:67: undefined reference to `vtable for std:🧵:_State'
 2021-05-05 11:35:27 +02:00
Johannes Schlatow
1c49da8ce4 terminal: don't overwrite last column until eol 
Introduce a new _overflowed state variable to indicate whether the
horizontal boundary was reached already and to omit subsequent character
output.

This state is necessary to maintain a valid cursor position at all
times. The _overflowed attribute is reset once the cursor is moved into
a valid position again.

To harmonise the bounds checking for _cursor_pos modifications, the
`constrain()` method was added.

Fixes genodelabs/genode#4093

Also fixes a bug in `vpa()` and `vpb()` which moved the cursor
horizontally instead of vertically.
 2021-05-05 11:35:27 +02:00
Norman Feske
9cb8c37e3a terminal: fix sporadic freeze on font-size changes 
The direct execution of application-level code by a watch handler is
troublesome because those handlers are executed at the I/O signal level.
In the concrete case, the watch handler got recursively called because
the handler called _handle_config (application-code), which called the
VFS, which in turn performed a wait_and_dispatch_one_io_signal, which
again invoked the watch handler.

The patch works around the problem by letting the watch notification
trigger the application-level signal for the _config_handler.

Fixes #4091
 2021-05-05 11:35:27 +02:00
Pirmin Duss
bb6617ad03 New libc_integration test 
I created a test program which stresses the interplay between libc,
pthreads and vfs_pipe and may detect regressions.

The program starts a thread that spawns a bunch of workers, sends and
receives data to them through a pipe. When all data of a worker is
collected, the worker is finished (join()) and a new worker is started.
Both the number of bytes sent to the worker and the size of its
answer are random.

Issue #3967
 2021-05-05 11:35:27 +02:00
Tomasz Gajewski
f49ec5b171 tool/run: extract function to copy files 
Issue #4073
 2021-05-05 11:32:09 +02:00
Tomasz Gajewski
507a7789fb tool/run: avoid error if init.xsd is missing 
Issue #4073
 2021-05-05 11:32:09 +02:00
Stefan Kalkowski
421d2bed40 platform driver: make device info XML optional 
* The device XML information dataspace is only provided,
  when the client's policy states `info="yes"`
* The device XM information gets changed to include the
  physical resource names (I/O memory and IRQ addresses)
  instead of virtual ids and page offset

Fix #4077
 2021-05-05 11:31:16 +02:00
Norman Feske
6caa74a18e Remove os/slave.h API 
Fixes #3754
 2021-05-05 11:21:43 +02:00
Norman Feske
5ac3c335dc Remove pkg/test-dynamic_config_slave 
Issue #3754
 2021-05-05 11:21:43 +02:00
Norman Feske
e8b97ad684 test/resource_yield: don't use os/slave.h 
Issue #3754
 2021-05-05 11:21:43 +02:00
Christian Helmuth
3f450a77e1 Fix touchscreen config in usb_hid README 2021-05-05 11:21:43 +02:00
Johannes Schlatow
462bff5aef dde_linux: test subdevice id for device matching 
As linux drivers may distinguish device configuration by pci subdevice id in
addition to the pci device id, the former must also be used for finding the
matching entry. Otherwise, e.g., the iwlwifi driver might load the wrong
firmware.

Side note: Add break statement to save superfluous iterations after
match was identified.

Fixes genodelabs#4076
 2021-05-05 11:21:43 +02:00
Norman Feske
ace7c9172b Disambiguate Mmio::local_addr 
This is a follow-up fix for "Streamline platform-device API on ARM".
There is an ambiguity of the 'local_addr' method between the inherited
'Attached_dataspace' and the local declaration, which results in the
double application of the sub-page Range::start.

Issue #4075
 2021-05-05 11:21:43 +02:00
Emery Hemingway
173264ed1e depot: remove ehmry 
Fixes #4082
 2021-05-05 11:21:43 +02:00
Christian Prochaska
190eafeaa6 base: handle 'unpack_string()' error in 'Arg::string()' 
Fixes #4100
 2021-05-05 11:21:43 +02:00
Josef Söntgen
ddf6a0c276 os: add invalid path exception to Path_base 
In case the path is invalid during import raise the 'Path_invalid'
exception.

Issue #4088
 2021-05-05 11:21:42 +02:00
Josef Söntgen
8f30fc993d libc: check path pointer in file I/O functions 
Fixes #4088
 2021-05-05 11:21:42 +02:00
Alexander Boettcher
33a64f79dc libc: place pthreads round-robin 
if the number of threads started exceed configured placement configuration

Issue #3967
 2021-05-05 11:21:42 +02:00
Pirmin Duss
7de62734e5 libc: add pipe2 symbol to symbols 
Issue #3967
 2021-05-05 11:21:42 +02:00
Norman Feske
d73eaaa14c libc: exclude arm_initfini.c 
This compilation unit contains a global constructor, which violates our
assumption that the libc is free of global constructors on ARMv7.
Specifically, the graphical terminal fails with the following message:

[init -> terminal] Error: Component::construct() returned without executing
                          pending static constructors (fix by calling
                          Genode::Env::exec_static_constructors())
[init -> terminal] Error: Uncaught exception of type 'Linker::Fatal'

In this case, the libc-less terminal uses the VFS. The VFS mounts the
ttf VFS plugin. The ttf VFS plugin depends on the libc.

The compilation unit 'arm_initfini.c' can safely be excluded because it
merely initializes the global '_libc_arm_fpu_present' variable, which
is not used by the current version of our libc/libm.

Fixes #4080
 2021-04-20 12:12:48 +02:00
Norman Feske
efbed6f7bf depot: update recipe hashes 2021-04-20 12:10:58 +02:00
Norman Feske
5ca024ff8b vfs_block.run: adjust cap quota for seL4 2021-04-20 12:10:58 +02:00
Norman Feske
dc8c899c1d Streamline platform-device API on ARM 
This API rework eases the access to memory-mapped I/O registers and
interrupts when using the platform driver. It introduces the notions of

- Platform::Device       - one device obtained from a platform session
- Platform::Device::Mmio - locally-mapped MMIO registers of a device
- Platform::Device::Irq  - interface for receiving device interrupts

The patch touches several drivers. Some drivers would require a
significant structural change to adopt the new API (e.g., net/virtio,
dde_linux drivers, imx gpio). In these cases, the patch adds
compatibility shims meant to be temporary. In other cases (e.g., imx
i2c), the adaptation was simple enough to carry through.

Fixes #4075
 2021-04-20 12:10:58 +02:00
Stefan Kalkowski
d1cf9c86b8 os: introduce device type in platform driver 
* Move "compatible" property value to "type" attribute
* Turn device_by_property helper in Platform::Connection into device_by_type

Fix #4074
 2021-04-20 12:10:58 +02:00
Norman Feske
46ca576eac Adjust gcov cap quota for seL4 2021-04-20 12:10:58 +02:00
Norman Feske
70281715c6 ssh_exec_channel.run: fix server IP pattern 
This patch adjusts the matching of the IP address such that the external
IP address is used as opposed the the NAT-local address. It is follow-up
fix of the conversion to the uplink session interface.

Issue #3961
 2021-04-20 12:10:58 +02:00
Norman Feske
4546148ab7 Adjust test-libc_fifo_pipe cap quota for seL4 2021-04-20 12:10:58 +02:00
Sebastian Sumpf
80cf47d906 ldso: protect object list with mutex 
When we allowed symbol resolution during exceptions, we used the shared
object lock to protect ELF object list manipulation (e.g., dlopen,
dclose) when executing exception unwinding code in the linker.
Unfortunately, sometimes libraries that are loaded by 'dlopen' may raise
exceptions in the process, leading to a deadlock within the unwind code.
In order to resolve this, we now protect the object list operations
(i.e., enqueue, removal, iteration) by a separate mutex. This allows
the shared object interface to throw exceptions.

issue #4071
 2021-04-20 12:10:58 +02:00
Norman Feske
7c01053842 Adjust test-fault_detection cap quota for seL4 2021-04-20 12:10:58 +02:00
Norman Feske
e28709d54d acpi_drv: use expanding reporter 
This makes the driver work on newer machines that produce reports larger
than 16 KiB. E.g., Sculpt OS boots successfully on a Lenovo P53s laptop
now.
 2021-04-20 12:10:58 +02:00
Norman Feske
dc89ebf978 core: kernel-agnostic 'Mapping' type 
This patch unifies the core-internal 'Mapping' type across all base
platforms.

As one minor downside on seL4, the diagnostic error messages when
observing faults other than page faults no longer print the faulting
thread and PD names.

Issue #2243
 2021-04-20 12:10:57 +02:00
Norman Feske
7ae1210531 fs_rom: remove "ROM file is missing" messages 
These messages pollute the boot log of Sculpt OS when ROM modules are
requested for files of the config fs before the sculpt manager has
created their first version.
 2021-04-20 12:10:57 +02:00
Norman Feske
c9d904df71 vfs/tar: remove "tar archive ..." message 
This diagnostic output is a mere debugging artifact. By removing them,
we make the boot of Sculpt less noisy.
 2021-04-20 12:10:57 +02:00
Norman Feske
21e9e1840a chroot: forward cap-quota upgrades to the server 
Fixes #4072
 2021-04-20 12:10:57 +02:00
Norman Feske
4e714d3f3a base: handle cap depletion during session upgrade 
When the own cap quota of a client does not suffice for a cap upgrade of
an existing session to a server, the client must issue a cap-resource
request to the parent. This logic was already in place for RAM quota but
was missing for cap quota.

Issue #4072
 2021-04-20 12:10:57 +02:00
Martin Stein
2084404aba nic_router: explain state report in README 2021-04-20 12:10:57 +02:00
Sebastian Sumpf
9c9302e51d ldso: allow symbol resolution during dl_itera_phdr 
When callback functions of `dl_iterate_phdr` required further jump slot
relocations this lead to a deadlock. Therefore, we allow the resolution
of further symbols from callback functions, but protect the ELF object
list during the iteration, which blocks any dynamic loading (e.g.,
dlopen/dlcose) of shared object by other threads while in program header
iteration.

fixes #4071
 2021-04-20 12:10:57 +02:00
Norman Feske
af490bdd5b Revert "sculpt: tweak window motions in leitzentrale" 
Apparently the skewed motion timings came from the issue fixed by
"timer: restore semantics of periodic timeout 0". With the current
version of the timer, the original motion parameters work just fine.
 2021-04-20 12:10:57 +02:00
Norman Feske
0339318572 platform_session: cache arg for alloc_dma_buffer 
This patch extends the 'Platform_session::alloc_dma_buffer' interface
with a 'Cache' argument that corresponds to the argument accepted by
'Ram_allocator::alloc', which is used by the platform driver under the
hood.

Since the x86 platform driver used to be hardwired to allocate DMA
buffers as UNCACHED, I adjusted all drivers by specifying the UNCACHED
argument. Right now, this is needed as a hint for core to steer the
allocation of I/O page tables. Once we eliminate the need for such hints
(by introducing an explicit 'Region_map::attach_dma' operation), we can
revisit the drivers individually because cached DMA buffers should
generally be fine on the x86 architecture.

Issue #2243
 2021-04-20 12:10:57 +02:00
Norman Feske
53e44f8bfd x86/platform_session: import Genode namespace 
This change avoids many repetetive Genode:: prefixes, making the code
easier to read. The patch also includes a few consistency fixes
regarding include guards and file headers. It also renames
Platform_device::String to Platform_device::Device:name.

Issue #2243
 2021-04-20 12:10:57 +02:00
Norman Feske
a839b4f0bb gdb_monitor: adapt to sandbox library changes 
Issue #4068
 2021-04-20 12:10:57 +02:00
Norman Feske
468e7a825c base/cache.h: rename Cache_attribute to Cache 
The short name is better because the type will become prominently
visible at the API.

Issue #2243
 2021-04-20 12:10:31 +02:00
Norman Feske
3ed8df9089 x86/platform_drv: add Platform::Session::dma_addr 
This patch adds the designated alternative to Dataspace::phys_addr to
the platform-session interface. Under the hood, the platform driver
still calls Dataspace::phys_addr but it should eventuelly become the
only caller before we can abolish this function.

Issue #2243
 2021-04-20 12:03:05 +02:00
Norman Feske
97a9ad114c platform_session: bus_addr_dma_buffer to dma_addr 
Issue #2243
 2021-04-20 12:03:04 +02:00
Norman Feske
f5f5b8c1f1 sandbox/init: route model 
This commit optimizes the 'Child::resolve_session_request'
implementation by introducing an internal 'Route_model' for quickly
traversing routing rules instead of parsing XML on each session request.

Fixes #4068
 2021-04-20 12:03:04 +02:00
Norman Feske
b661459aca sandbox/init: parse config using 'List_model' 
This commit replaces the hand-crafted config processing by the use of
the 'List_model' utility. This has the following advantages:

- The parsing follows a common formalism that makes the code
  easier to maintain and to understand. Several parts of the code
  had to be changed (for the better) to make it fit the list model
  approach. E.g., the child states have become more expressive
  and logical.

- In the common case, the XML data is traversed only once, which
  increases the parsing speed in dynamic scenarios.

- The code becomes easier to optimize. In particular, the patch skips
  the re-evaluation of the session routing if no service is affected
  by the config change.

The patch also revisits the init test by removing overly long sleep
phases and extending a few sleep phases that were too short when
executing the test on Qemu.

Issue #4068
 2021-04-20 12:03:04 +02:00
Norman Feske
f925fef17b base: Refine Range_allocator::alloc_aligned 
This patch changes the 'alloc_aligned' interface as follows:

- The former 'from' and 'to' arguments are replaced by a single
  'range' argument.

- The distinction of the use cases of regular allocations vs.
  address-constrained allocations is now overed by a dedicated
  overload instead of relying on a default argument.

- The 'align' argument has been changed from 'int' to 'unsigned'
  to be better compatible with 'addr_t' and 'size_t'.

Fixes #4067
 2021-04-20 12:03:04 +02:00
Tomasz Gajewski
97d44c5a79 app/menu_view: use CUSTOM_TARGET_DEPS 2021-04-20 12:03:04 +02:00
Tomasz Gajewski
1867cf4967 iwl_firmware: use CUSTOM_TARGET_DEPS 2021-04-20 12:03:04 +02:00
Tomasz Gajewski
f3f8d9a6de build system: support for CUSTOM_TARGET_DEPS 2021-04-20 12:03:03 +02:00
Norman Feske
99f4b3cd07 Debug macros for easy performance profiling 
The debug macros 'GENODE_LOG_TSC' and 'GENODE_LOG_TSC_NAMED' allow for
the easy gathering of the TSC ticks consumed by the calling scope.

Fixes #4066
 2021-04-20 12:03:03 +02:00
Norman Feske
ac07f9d08e sculpt: skip GUI handling until framebuffer is up 
This avoids unneeded activity of the Leitzentrale GUI (e.g., the
decorator) at the early boot phase of Sculpt.
 2021-04-20 12:03:03 +02:00
Norman Feske
ed9487b452 rtc_drv: defer init until first use 
This patch avoids the RTC driver's aggressive polling during the
(CPU-bounded) boot of Sculpt OS. As the RTC is not needed during bootup,
this patch reduces the boot time (on Qemu) by 2-4 seconds.

Fixes #4065
 2021-04-20 12:03:03 +02:00
Norman Feske
729b22f04f rtc_drv/x86: code structure 
Give the driver code a proper context in the form of a 'Driver' class.

Adjusted while investigating issue #4065.
 2021-04-20 12:03:03 +02:00
Norman Feske
698f6eb86c gdb_monitor: weaken reliance on sandbox internals 
GDB monitor re-uses internal parts of the sandbox library. It thereby
relies on the internal detail of how local headers are included (using
"" instead of the now usual practice <>). A seemingly unrelated change
"init/sandbox: avoid repetitive state reports" eventually broke the
build of the GDB monitor.

This patch makes the GDB monitor less reliant on the way of how the
sandbox includes headers internally.

It also fixes a few warnings caused by the double definition of
__GENODE__ (adding an missing include guard and making the #define
conditional).

Related to issue #4064
 2021-04-20 12:03:03 +02:00
Norman Feske
73e8d64c34 init/sandbox: avoid repetitive state reports 
Fixes #4064
 2021-04-20 12:03:03 +02:00
Christian Helmuth
cb9a26f1ed vbox6: sculpt runtime package 
Issue #4031
 2021-04-20 12:03:03 +02:00
Norman Feske
49481dd3fb timer: restore semantics of periodic timeout 0 
The 'Timer::Session::trigger_periodic' RPC function used to accept 0 as
a way to de-schedule the periodic processing. Several components such as
nitpicker relied on this special case. In "timeout: rework timeout
framework", the value of zero was silently clamped to 1, which has the
opposite effect: triggering signals at the maximum rate. This results in
a visible effect in Sculpt where the leitzentrale-nitpicker instance
produces a constant load of 2% CPU time.

This patch restores the original timer semantics by

- Documenting it in timer_session.h,

- Handling the case explicitly in the timer implementation, and

- Replacing the silent clamping of the unexpected value 0 passed
  to the timeout framework by a diagnostic error message.

Issue #3884
 2021-04-20 12:03:03 +02:00
Norman Feske
dc8dd3396d base: move Xml_generator::Node::Node behind ABI 
Fixes #4063
 2021-04-20 12:03:03 +02:00
Christian Helmuth
9ac23a18d4 vbox6: synchronize guest-memory handling 2021-04-20 12:03:03 +02:00
Norman Feske
ae8050bb82 wm.run: let nitpicker request fb and input 
This patch restores the ability to launch nitpicker within a gui_fb
window. It is a follow-up fix for "nitpicker: make framebuffer and input
optional".

Issue #3812
 2021-04-20 12:03:02 +02:00
Tomasz Gajewski
0a849a1681 prepare_port: cache handling for files with paths 
This patch is a follow-up fix for issue #4028 that fixes #4061.
 2021-04-20 12:03:02 +02:00
Christian Prochaska
f89414c637 libports: update expat port 
Fixes #4060
 2021-04-20 12:03:02 +02:00
Christian Helmuth
4cdfeb13e2 vbox6: cleanup code and log output 
Issue #4031
 2021-04-20 12:03:02 +02:00
Christian Helmuth
1a57a5a959 vbox6: refactor vcpu implementation 
Also add recent SVM changes from vbox5.

Issue #4031
 2021-04-20 12:03:02 +02:00
Norman Feske
96cc660f95 sculpt: system reset via PS/2 and ACPI 
This patch enables sculpt to utilize the CPU reset mechanism via the
PS/2 controller as well as the information provided via the ACPI FADT
information. Whenever the /config/system file is changed to <system
state="reset"/>, both mechanisms are triggered.

Supporting both mechanisms is useful because the PS/2-based reset does
not work reliably on modern machines. The PS/2-based reset is useful in
the case when the FADT reset information refers to the PS/2 command
port. In this case, the platform driver is unable to access this port
because it is already handed out to the PS/2 driver. In this case, the
PS/2 driver kicks in.

Issue #2726
 2021-04-20 12:03:02 +02:00
Norman Feske
2c7c7767fc platform_drv: respond to <system state="reset"/> 
If 'system="yes"' is specified in the <config>, the platform driver now
monitors the content of the system ROM. Should the value become "reset",
a reset is triggered using information obtained from the ACPI driver.

Related to issue #2726
 2021-04-20 12:03:02 +02:00
Norman Feske
9f8c555e7d ps2_drv: respond to <system state="reset"/> 
This patch adds the option for monitoring a system ROM to the PS/2
driver and issuing a CPU reset whenever the state becomes "reset".

Issue #2726
 2021-04-20 12:03:02 +02:00
Norman Feske
fb9f6812e1 acpi_drv: export ACPI reset info from FADT 
The change of the platform driver is needed to avoid a parsing error of
the ACPI ROM when encountering an unexpected <reset> node.

Thanks to Alexander Boettcher for the preventing the regression of #3829
and improving the adherence to the ACPI spec when parsing the reset type.

Related to issue #2726
 2021-04-20 12:00:09 +02:00
Christian Helmuth
839183d2b6 vbox6: unify EMT and vCPU entrypoint 
Issue #4031
 2021-04-20 11:57:19 +02:00
Christian Helmuth
c67e78a7f0 vbox6: update to version 6.1.18 
Issue #4031
 2021-03-29 15:49:59 +02:00
Christian Helmuth
0ac4d1d411 libc: rename special-purpose pthread creation functions 
The functions are used by our VirtualBox ports to hook into thread
creation and, thus, are listed in the ABI symbols.

Issue #4031
 2021-03-29 15:48:11 +02:00
Norman Feske
0f9cb72cfa depot: update recipe hashes 2021-03-29 13:46:49 +02:00
Norman Feske
27527bf165 wm: allow clients to upgrade session caps 
This patch adds the handling neccessary for clients to upgrade the cap
quota of their GUI session. Until now, the upgrade mechanism supported
only RAM quota.

The limitation became visible once a client - in the particular case the
motif_decorator - attempted the cap upgrade of its GUI session. The cap
quota would arrive at the wm, but the wm would keep it instead of
forwarding the quota to nitpicker. This resulted in an infinite retry
loop at the motif_decorator, ultimatedly depleting all its cap quota.
 2021-03-29 13:45:25 +02:00
Norman Feske
f839b3ecba wm: make hover handling robust against input races 
The window manager infers the overall state from the intercepted input
events for the decorator and all GUI clients. However, each of those
parties have an independent input-event stream. Whereas the order of
events within one GUI session is strict, the order of events between GUI
sessions is arbitrary. The window manager wrongly relied on a global
event ordering to track the pointed-at GUI session.

The patch removes the assumption of a global event order by tracking the
relevant pointer state for each GUI session independently and evaluating
these states when propagating the pointer position to the decorator.

Fixes #4059
 2021-03-29 13:45:24 +02:00
Norman Feske
bfea27a258 Tweak to support delete key in bash 
This commit modified bash's builtin mapping of character sequences to
readline-functionality to detect the sequence of the delete symbol as
generated by the terminal.

Fixes #4058
 2021-03-29 13:45:24 +02:00
Christian Helmuth
4f91d71cf9 zynq_qemu: disable fb_bench and nic_router_flood tests 2021-03-29 13:45:24 +02:00
Jean-Adrien DOMAGE
32169cd137 add I2c driver for imx8q based platform 
This commit introduces a new i2c driver for the imx8q_evk based platform.

Fixes #4052
 2021-03-29 13:45:24 +02:00
Norman Feske
eb89b13327 depot: update recipe hashes 2021-03-24 11:32:53 +01:00
Norman Feske
b51c1a0fe3 News item for Sculpt 21.03 2021-03-24 11:32:53 +01:00
Norman Feske
f90cd542cb sculpt: documentation update 2021-03-24 11:32:53 +01:00
Christian Helmuth
dce272ba8f tool/xkb2ifcfg: update README to event_filter 2021-03-24 11:09:08 +01:00
Alexander Boettcher
141af733aa vbox5: remove priority bands from vbox5-* packages 2021-03-24 10:34:55 +01:00
Alexander Boettcher
945b4760ef sculpt: remove vfs.lib and init from common rules 
to support packages using own build versions and to detect packages missing
to specify dependency on init.
 2021-03-24 10:34:55 +01:00
Alexander Boettcher
53041f4cd8 usb_host: support to disable usb controller types 
The commits adds the same configuration values as supported up to now by
the monolithic usb driver. In contrast to the original, by default all drivers
are started. Disabling a controller type is used by Sculpt, e.g. for OHCI if
running Sculpt inside Virtualbox.
 2021-03-23 11:50:37 +01:00
Alexander Boettcher
521f61b9e0 usb_hid: disable wacom driver 
Issue #3997
 2021-03-23 11:32:24 +01:00
Norman Feske
ca50a41d28 depot: update recipe hashes 2021-03-23 11:28:53 +01:00
Norman Feske
b29f1497bf depot_query: sort depot-user scan results 
Related to issue #4054
 2021-03-23 11:28:21 +01:00
Norman Feske
ca5522d4d9 sculpt: move chargen files to config/keyboard/ 
The addition of further keyboard layouts would otherwise pollute the
config/ directory too much.

Caution: When updating Sculpt OS, don't reuse your existing
config/event_filter file unmodified! You need to replace the chargen
includes, pointing to the new location, e.g.,

  <include rom="en_us.chargen"/>
  <include rom="special.chargen"/>

must be changed to

  <include rom="keyboard/en_us"/>
  <include rom="keyboard/special"/>

Issue #4055
 2021-03-23 11:28:21 +01:00
Norman Feske
36ef41626a sculpt: keyboard-layout selection dialog 
This patch extends the settings dialog with the ability to select the
keyboard layout between the options that are included in the sculpt
image. The manual configuration is of course still possible by editing
the /config/event_filter directly.

If both the fonts configuration and the event-filter configuration are
managed manually, the settings button and window are not displayed.

Fixes #4055
 2021-03-23 11:28:21 +01:00
Norman Feske
e9ac14ed49 fs_query: report dir content in alphabetic order 
By sorting the reported output, all consumers of the reports become able
to rely on a deterministic order. For example, the file browser of
Sculpt OS, the launcher menu items, and the depot-selection items will
appear in a predictable order.

Fixes #4054
 2021-03-23 11:28:20 +01:00
Norman Feske
8f1db47c26 sculpt: screen capturing and event injection 
This patch adds session-routing options to connect components to the
capture/event interfaces of the global nitpicker GUI server or the
leitzentrale GUI server. It thereby enables the implementation of screen
capturing components, remote management tools, or virtual keyboards.

Since those services are very powerful, they are subsumed under the
"hardware" category of the component graph as opposed to the GUI
category, which guarantees the separation of clients.

Fixes #4053
 2021-03-23 11:28:20 +01:00
Norman Feske
d2fc834bfa os: depot recipe for pkg/test-capture 
This package allows for experimenting with the capture-session
interface.
 2021-03-18 17:36:35 +01:00
Christian Prochaska
3d432331b9 libc: mmap: use VFS handle as reference for shared mappings 
Fixes #4050
 2021-03-17 16:30:41 +01:00
Christian Prochaska
446df00d0d libc: support 'RLIMIT_DATA' resource in 'getrlimit()' 
Fixes #4049
 2021-03-16 17:45:40 +01:00
Tomasz Gajewski
2f0898d2a9 dde_linux: fixes in makefiles 
Issue #3972
 2021-03-16 11:25:25 +01:00
Christian Prochaska
9a0217f21a imx8: use 'snps,dis_u2_susphy_quirk' 
Fixes #4048
 2021-03-16 11:19:32 +01:00
Christian Prochaska
0cfafa1c8f usb_host_drv: support command timeouts and bool properties 
Fixes #4047
 2021-03-16 11:19:17 +01:00
Norman Feske
2c85e48a0d Revert "run: tweak CRLF translation in log scripts" 
The substitution eats carriage-return characters in some cases such as
the Pine-A64-LTS board connected via an TTL-232R-RPi cable.
 2021-03-12 15:08:11 +01:00
Norman Feske
15780a657c sd_card_bench.run: allow timer to reach IO_MEM 
This is needed on seL4 on i.MX6 where the user-level timer driver
accesses memory-mapped I/O registers directly.
 2021-03-12 12:09:17 +01:00
Norman Feske
5c5b56d1e0 depot: update recipe hashes 2021-03-12 12:08:24 +01:00
Norman Feske
00900d82b5 sculpt: tweak window motions in leitzentrale 2021-03-12 12:01:39 +01:00
Alexander Boettcher
18182b11da base: exit loop on cpu session close 
if session can't be closed immediately, e.g. if the service is provided by
a child (cpu_balancer) and not by the parent.

Issue #4029
 2021-03-12 12:01:39 +01:00
Norman Feske
8eb514d6b5 sculpt: interactive priority configuration 
This patch reworks the resource-assignment dialog to allow the user to
select the scheduling priority in addition to the CPU affinity.

Fixes #4045
 2021-03-12 12:01:39 +01:00
Alexander Boettcher
8a8de970a5 sculpt: add system clock package to index 
Issue #4046
 2021-03-12 12:01:39 +01:00
Sid Hussmann
cae3e447d6 vfs/pipe: fix pagefault during congested pipes 
Issue #4030
 2021-03-12 12:01:39 +01:00
Alexander Boettcher
f98d10a3f3 depot_deploy: support packages providing Rtc 
Fixes #4046
 2021-03-12 12:01:39 +01:00
Alexander Boettcher
521663c6de system_rtc: add standalone runtime package 
Package contains no rtc driver, which must be provided separately.

Issue #4046
 2021-03-12 12:01:39 +01:00
Alexander Boettcher
9b5bedefc7 expanding_pd_session: remove cap warning message 
which are caused often when a component runs a bit longer and has dynamic
resource usage, e.g. running Genode's tool chain.
 2021-03-12 12:01:39 +01:00
Norman Feske
8ecc258d3f sculpt: introduce runtime priorities 
This patch adds 4 priority levels to the runtime subsystem. The highest
priority is used for components that are critical for the operation of
Sculpt, in particular the Leitzentrale GUI. All regularly deployed
components are assigned the lowest priority by default.

With priorities available in the runtime subsystem, this patch flattens
the priority levels at the top-level init to only two levels and
overlays the priority bands of the drivers, leitzentrale, and runtime
subsystems into one priority band. This has three benenfits:

- This change prevents the starvation of the Leitzentrale GUI from a
  spinning high-priority driver (issue #3997).

- The change will also ease the hosting of latency-critical components
  in the runtime subsystem that are prioritized higher than regular
  components, the storage stack, and the network stack.

- The Leitzentrale GUI remains always perfectly responsive regardless
  of the workloads deployed from packages. In the previous version,
  the runtime graph was sometimes stuttering on high system load.

Issue #4045
 2021-03-12 12:01:39 +01:00
Sebastian Sumpf
7bbd050f25 riscv: adjust timeout run script to RISC-V 
* Limit required memory for RISC-V to 15 MB per buffer

issue #4021
 2021-03-12 12:01:39 +01:00
Norman Feske
7e7c10e66c Add sandbox test to depot autopilot 
Issue #4044
 2021-03-12 12:01:39 +01:00
Norman Feske
d5d3b3c3a4 test.run: support binaries other than init 
This change enables the use of runtime pkgs that specify the test binary
directly as attribute in the <runtime> node.
 2021-03-12 12:01:39 +01:00
Norman Feske
2baa283d87 vfs_lwip: reduce repeated warnings 
This patch avoids the repeated warning "read blocked until lwIP
interface is ready" by printing the message only once. Otherwise,
the log is flooded with those warnings when falkon web browser
is started on Sculpt OS without network connectivity.
 2021-03-12 12:01:39 +01:00
Josef Söntgen
4a12b5c653 mixer: fix README with regard to boolean values 
Issue #4043.
 2021-03-12 12:01:39 +01:00
Josef Söntgen
ba6c4a664f mixer: use audible default settings 
In case there is no valid configuration the component used to mute
output by default. This, however, might lead to the assumption that
audio is not working at all. Instead set the master output volume to
50% and the per application volume to 100%.

Fixes #4043.
 2021-03-12 12:01:38 +01:00
Norman Feske
9093c293cb sandbox: destroy 'Session_state' of local services 
This patch adds the missing destruction of session-state objects of
local services when closing a session. Because of the missing
destruction, those session-state object remained part of the server
ID space. This becomes a problem once the backing store of the session
state object vanishes, that is when the client child gets removed from
the sandbox. Hence, the removal of a child with an open session to a
local service would lead to the corruption of the server ID space.

This patch adds the missing session.destroy() call.

Fixes #4044
 2021-03-12 12:01:38 +01:00
Norman Feske
935bb36fe4 base: fix child destruction while close requested 
This patch fixes a corner case where a child is destructed while a
asynchronous close request to a sibling server is still pending.

The child immediately discarded the session ID as the end of the
close-session processing, assuming that this ID is never to be needed
again. The session-state continues to exist to handle asynchrous close
protocol with the server.

However, if the child is destructed at this point (before the server
responded to the session request), the destruction of the child would
not cover the discharging of the session state because the session state
was no longer be part of the client's ID space. So once the asynchronous
close response from the server came in, the session state contained
stale information, in particular a stale closed_callback pointer.

The patch fixes the problem by deferring the discarding of the client ID
to the point where the session state is actually destructed. So the
session of a pending close response is covered by the child destructor.

Thanks to Pirmin Duss for reporting this issue along with a test
scenario for reproducing it!

Fixes #4039
 2021-03-12 12:01:38 +01:00
Norman Feske
755aed7cb2 dde_linux: don't spin when panicking 
Issue #3997
 2021-03-12 12:01:38 +01:00
Sebastian Sumpf
6223ae4413 riscv: Interrupt controller support 
- Enable the "platform-level interrupt controller" PLIC on base-hw
- The RISC-V specification offers only a register description, but no
  layout for the register set. This implies the layout is platform
  dependent, and therefore, implemented separately for Qemu

issue #4042
 2021-03-12 12:01:38 +01:00
Sebastian Sumpf
bebba3876e run boot_dir hw: build debug ELF binaries again 
- build debug version of core and bootstrap found under
  _var/run/<runscript>.core_ (.bootstrap). This is a regression when
  switching from object to archive library linkage of core.

fixes #4027
 2021-03-12 12:01:38 +01:00
Norman Feske
aa0a98bd43 base-foc/hw: avoid use of placement new operator 
Fixes #2106
 2021-03-12 12:01:38 +01:00
Josef Söntgen
42f3d2eccd qemu-usb: reset completion pointer 
Reset the completion pointer to prevent a double free in case the
packet is processed during ISOC IN request handling.

Fixes #4041.
 2021-03-12 12:01:38 +01:00
Josef Söntgen
c03534e355 boot_dir/nova: enable intel_hwp 
Instruct bender to enable Intel HWP on supported CPUs unconditionally
to gain more insight by using the feature most prominently in Sculpt.

Fixes #4040.
 2021-03-12 12:01:38 +01:00
Norman Feske
1e0d843464 libc: avoid using placement new operator 
Issue #2106
 2021-03-12 12:01:38 +01:00
Norman Feske
8c7d34ff21 rpi_platform_drv: remove placement new operator 
Issue #2106
 2021-03-12 12:01:38 +01:00
Norman Feske
d6a312f438 tool/builddir: pass --ccache to run tool 
With this patch, the build directory's CCACHE configuration is applied
to the building of depot archives when using the --depot-auto-update
feature.

Note that this is effective only when resetting the hash files before
building so that each depot-auto-update results in the same-named
versions. Otherwise, the include paths (that include the version name as
part of the path) make cache hits impossible.

Issue #4004
 2021-03-12 12:01:38 +01:00
Norman Feske
6544cca320 tool/run: add --ccache argument 
If this argument is combined with --depot-auto-update, the run tool
enables the use of ccache when invoking the depot/create tool.

Issue #4004
 2021-03-12 12:01:38 +01:00
Norman Feske
3d0ed5992d tool/depot/create: add CCACHE=1 option 
This commit complements issue #4004 with support for the depot/create
tool. If CCACHE=1 is specified at the command line, the build directory
for building a binary archive is configured with 'CCACHE := yes'.
 2021-03-12 12:01:38 +01:00
Norman Feske
366fda0e47 sculpt: release part_block after device discovery 
This patch triggers the immediate removal of part_block once the
discovery of a fresh inserted USB stick has completed. Without the
patch, part_block was retained until the runtime was reconfigured the
next time for other reasons (e.g., toggling an inspect view). Until the
next reconfituration, part_block tended to stand in the way of directly
assigning the USB device to a VM.
 2021-03-12 12:01:38 +01:00
Christian Helmuth
7ce1f8e92d Enable en_us chargen in drivers_interactive-imx8q_evk 
Issue #4038
 2021-03-12 12:01:37 +01:00
Christian Helmuth
6e9843bd05 test-libc_fifo_pipe: more cap quota required for sel4/x86_64 
Issue #4030
 2021-03-12 12:01:37 +01:00
Norman Feske
2ff252360d sculpt: avoid flickering of leitzentrale 
This patch increases the RAM quota of the nitpicker instance for the
leitzentrale to make it suffice for the buffering of content during
resize operations. This fixes a flickering artifact when having an
inspect view open while entering a WLAN passphrase. When the connect
button appears or disappears (depending on the number of written
characters), some parts of the inspect terminal would flicker sometimes.
 2021-03-12 12:01:37 +01:00
Norman Feske
9de61e7014 base: detect truncated session args 
This patch reflects the truncation of session arguments (e.g., by very
long session labels) by denying the session request.

Fixes #4037
 2021-03-12 12:01:37 +01:00
Norman Feske
6712eac7e6 platform driver: use same label for all device PDs 
Fixes #4037
 2021-03-12 12:01:37 +01:00
Martin Stein
25a212aa24 nic_router: fix uncaught bit alloc exception 
* catch every possible path of the Bit_allocator_dynamic::Out_of_indices
  exception
* add unconditional log output in the new "catch" directives, so, we will be
  able to debug the problem a bit more in detail next time

Fixes #4036
 2021-03-12 12:01:37 +01:00
Alexander Boettcher
89ffc48576 depot_deploy: integrate shim and adapt sculpt 
Issue #3837
Issue #4029
 2021-03-12 12:01:37 +01:00
Norman Feske
9a5bc9caf0 cpu_balancer.run: use shim for hosting cpu_burner 
This way, the redirected env sessions for the CPU burner are no longer
called directly by init. Init stays independent from the behavior of the
CPU balancer.

Issue #3837
Issue #4029
 2021-03-12 12:01:37 +01:00
Norman Feske
c0a7565c21 os: shim helper for safe routing of env sessions 
This shim component can be used in case where env sessions of child
(i.e., child's PD session) must be routed to another child of init.

Without the shim, init would directly need to interact with these
sessions and would thereby make itself dependent of the server's
behavior. RPC calls to a server hosted as a child lead to all kinds of
problems such as livelock situations, and putting the robustness of init
at the whim of its child.

With the shim, init merely needs to bootstrap the shim component by
routing the shim's env sessions to core as usual. The server is only
used for the sessions for the actual application hosted atop the shim.

Issue #3837
Issue #4029
 2021-03-12 12:01:37 +01:00
Alexander Boettcher
a02ec07e49 sculpt_manager: add PD as service 
Issue #3837
 2021-03-12 12:01:37 +01:00
Alexander Boettcher
1f29055927 depot_deploy: support providing CPU and PD service 
Issue #3837
 2021-03-12 12:01:37 +01:00
Alexander Boettcher
7af276ac81 cpu_balancer: add support to provide PD service 
Issue #3837
 2021-03-12 12:01:37 +01:00
Tomasz Gajewski
de62582905 prepare_port: simple cache for downloads 
Fixes #4028
 2021-03-12 12:01:37 +01:00
Norman Feske
ba567f4ba8 Avoid double slashes in select_from_repositories 
Issue #4019
 2021-03-12 12:01:37 +01:00
Alexander Boettcher
ee0ed273e6 vm_stress: increase cap quota for 3 & 4 vcpu test 
due to changes by

   base: refactor VM/vCPU API

   Issue #3554
 2021-03-12 12:01:37 +01:00
Norman Feske
e1bb0e8e15 usb_host: refine class heuristics on device report 
This patch is an interim fix for using HID devices that offer a HID
interface as not the first interface. It also supplements the
interface classes as supplemental information to the USB-devices
report.

Fixes #4035
 2021-03-12 12:01:37 +01:00
Norman Feske
2e4ccc1459 usb_host: reflect ENOSPC as STALL 
Issue #4035
 2021-03-12 12:01:37 +01:00
Norman Feske
80522fadf6 usb_hid: don't touch non-HID interfaces 
Issue #4035
 2021-03-12 12:01:36 +01:00
Norman Feske
2ce4a3b400 sculpt: version 21.03 2021-03-12 12:01:36 +01:00
Norman Feske
c68443e2eb base-hw: support packaging of multi-arch boards 
With this patch, the board/<name>/arch property file can have multiple
lines where each line denotes an instruction-set architecture supported
for the board.

Issue #4019
Fixes #4034
 2021-03-12 12:01:36 +01:00
Stefan Kalkowski
9685a8b60d drivers_managed-imx8q_evk: add dummy_rtc_drv 
Ref #4033
 2021-03-12 12:01:36 +01:00
Tomasz Gajewski
23e3079f46 tool/run: option to copy tftp served files 
Fixes #3386
 2021-03-12 12:01:36 +01:00
Stefan Kalkowski
10b56afff0 rtc_drv: add dummy driver 
* Add dummy driver implementation and depot package
* Extend configuration to allow setting an initial RTC value

Fix #4033
 2021-03-12 12:01:36 +01:00
Sebastian Sumpf
d4b58b689c base-hw: fix RISC-V duration calculation 
Simplify calculation of Timer::_duration, the old implementation caused
the time running backwards sometimes. This makes
'nic_router_dhcp_*' and 'event_filter' run scripts succeed.

issue #4021
 2021-03-12 12:01:36 +01:00
Norman Feske
1826ff8a59 sculpt: constrain initial log-window size 
This prevents the log terminal from adjusting itself to the whole screen
size before the leitzentrale window layout is applied. This intermediate
size can otherwise exhaust the 8 MiB of RAM quota of the log terminal.

Issue #3970
 2021-03-12 12:01:31 +01:00
Christian Helmuth
86ad4ed17f version: 21.02 2021-02-25 11:26:18 +01:00
Norman Feske
1d1b5b88c5 News item for version 21.02 2021-02-25 11:24:28 +01:00
Norman Feske
4f1a3a8000 Release notes for version 21.02 2021-02-25 11:24:28 +01:00
Christian Helmuth
0afd3db894 depot: update recipe hashes 2021-02-25 11:24:23 +01:00
Christian Helmuth
cbe81d35b9 autopilot: disable run script not supported by riscv 
Issue #4021
 2021-02-25 11:22:59 +01:00
Josef Söntgen
1d551bd967 vbox6: add QEMU xHCI device-model 2021-02-25 11:22:59 +01:00
Alexander Boettcher
812c3599de vbox6: add network backend using the nic session 
as used in vbox5 already.
 2021-02-25 11:22:59 +01:00
Christian Helmuth
20caac5f3b Port of VirtualBox 6.1.14 2021-02-25 11:22:59 +01:00
Christian Helmuth
a47b374905 vim.run: import clipboard archive only once 2021-02-23 13:22:50 +01:00
Christian Helmuth
7a3dc68f34 vbox5: fix typo 2021-02-23 13:22:48 +01:00
Christian Prochaska
dd92ab126b libc: test TLS support in pthread test 
Fixes #4023
 2021-02-23 12:19:25 +01:00
Sebastian Sumpf
f68e655312 mk: link core to library archive instead of .o file 
Do not link base and core libraries into on large relocatable .o file,
which is linked later to core - causing long link times. Create an
independent library archive out of the base and core libraries that can
be linked faster.

issue #4027
 2021-02-23 12:17:04 +01:00
Norman Feske
64165d829e depot: update recipe hashes 2021-02-23 12:07:18 +01:00
Norman Feske
c2feba065f Reduce pointers to doc/ in top-level README 
Avoid referencing the content of doc/ for topics that are covered by the
Genode Foundations book.
 2021-02-23 12:07:18 +01:00
Christian Helmuth
219809ffed base: refactor VM/vCPU API 
Issue #3554
 2021-02-23 12:07:18 +01:00
Sid Hussmann
6e8728f2d3 vfs/pipe: use local signal handling 
This commit improves the performance of the pipe plugin by using local
signal handling and avoiding sending a signal during write when pipe
buffer is full.

Issue #3583
 2021-02-23 12:07:17 +01:00
Stefan Kalkowski
90d9470dfd vmm: add virtio block device model 
* Add new virtio device model
* Extend test run-script with vfat block test image
* Add vmm depot src recipe
* Use packages in test run-script

Fix #4025
 2021-02-23 12:07:17 +01:00
Stefan Kalkowski
2879aa003b vmm: cleanup virtio model, make it extensible 
* Introduce different index types for ring counters and descriptor arrays
  within the Virtio split queue implementation
* Be more accurate in reporting the queue number supported, and raise it
  to 512
* Introduce abstractions for mmio register access, where several values
  are stored at the same place, and selector registers exist.
* Turn Virtio_device into a template to define the Virio queue type,
  and its numbers per device model (needed for e.g., future block model)

Issue #4025
 2021-02-23 12:07:17 +01:00
Stefan Kalkowski
83c2309710 run: don't rebuild core/bootstrap when available 
Fix #4026
 2021-02-23 12:07:17 +01:00
Christian Prochaska
59459e60e7 libc: pthread TLS optimizations 
Fixes #4024
 2021-02-23 12:07:17 +01:00
Sid Hussmann
8d13121e84 vfs/pipe: refactor read/write synchronization 
All pipe-ends were notified at the same time which leads to dead-locks.
This commit mitigates this by having a signal handler for each pipe and
each pipe-end respectively.

Issue #3583
 2021-02-23 12:07:17 +01:00
Sid Hussmann
3ff0efd627 vfs/pipe: add fifo feature to pipe plugin 
The vfs pipe plugin can now be used as named pipe which anables data
transfer via file handles from one component to another. E.g. if one
would like to send data from component A to stdin of a libc component B,
one can do so by simply writing to that fifo file.

Issue #3583
 2021-02-23 12:07:16 +01:00
Norman Feske
10605a6903 Spelling fix 2021-02-23 12:07:16 +01:00
Martin Stein
6937eb7d94 drivers_nic-rpi: use usb_net and usb_host 
Ref #3961
 2021-02-23 12:07:16 +01:00
Martin Stein
a462a8e741 usb_host: compile errors with DEBUG_LINUX_PRINTK 
Ref #3961
 2021-02-23 12:07:16 +01:00
Martin Stein
3485282909 drivers/platform/rpi: remove dead code 
Ref #3961
 2021-02-23 12:07:16 +01:00
Sebastian Sumpf
b6d20b4742 autopilot: disable run script not supported by riscv 
- libc and nic driver are currently not available on RISC-V

issue #4021
 2021-02-23 12:07:16 +01:00
Sebastian Sumpf
7318ca6084 depot: add 'riscv' to depot spec 
issue #4021
 2021-02-23 12:07:16 +01:00
Sebastian Sumpf
ca777fe93f riscv: add riscv_qemu base-hw recipe 
issue #4021
 2021-02-23 12:07:16 +01:00
Sebastian Sumpf
ccd9ba4161 riscv: move CPU header from base-hw to base 
issue #4021
 2021-02-23 12:07:16 +01:00
Christian Helmuth
954f03257d Update curl to version 7.70.0 
Issue #3773
 2021-02-23 12:07:16 +01:00
Pirmin Duss
190b4784c5 openssl: do not depend on ARM CPU probing 
With this commit libcrypto does not use ARM NEON extension as long as
SPECS includes "neon". arm_v7a does declare "neon" per default while
arm_v8a does.

Issue #3773
 2021-02-23 12:07:16 +01:00
Christian Helmuth
f23e302475 Update OpenSSL to version 1.1.1i 
Note, OpenSSL now comes as one combined depot archive *openssl* that
replaces the former *libssl* and *libcrypto* archives. The libraries are
still separate binaries for compatibility with legacy software.

Issue #3773
 2021-02-23 12:07:16 +01:00
Sebastian Sumpf
f5cd12dcf9 base/base-hw: unify startup library builds 
- Use 'select_from_repositories' for crt0.s of startup library
- Move RISC-V crt0.s to base (where the other archs are located)

issue #4019
 2021-02-23 12:07:15 +01:00
Martin Stein
ce31c90bc3 run/netperf.inc: fix problem with reused ports 
It seems that with the nigthly autopilot the data port (that is choosen by the
client) may stay some time (minutes) allocated after the a test run. This
causes successive tests to fail when trying to reuse the port.

* Use a unique data port for each target platform in the range of 18000-18099.
* Add documentation on how to prepare for running the test on a Linux target.
* Remove Qemu-specific code paths as Qemu isn't supported anymore by the script
  since the introduction of the Uplink session (running with a NIC router on
  Qemu was never supported and now we always need a NIC router).

Ref #3961
 2021-02-23 12:02:46 +01:00
Martin Stein
f9c258a372 lwip_lx.run: typo in doc 
Ref #3961
 2021-02-23 12:02:45 +01:00
Norman Feske
048a4625c5 base: remove stale spec/fpu_vfpv3.mk file 2021-02-23 12:02:45 +01:00
Norman Feske
db3f86d603 base-hw: avoid double slashes in path names 2021-02-23 12:02:45 +01:00
Norman Feske
fa68325a57 base-hw: streamline extraction of depot archives 
This patch simplifies the packaging of the base-hw kernel for a given
board, in particular when the board support is hosted as an external
repository such as genode-world. Regardless of where the board-support
is hosted, the content.mk file becomes as simple as:

  include $(GENODE_DIR)/repos/base-hw/recipes/src/base-hw_content.inc

The board name is automatically inferred from the path of the src
recipe. The architecture is determined from board/<name>/arch files,
following the pattern of the image_link_address. The attempt to build a
base-hw-<board> binary archive for the wrong architecture is now
gracefully handled by skipping all targets (using the REQUIRES
mechanism).

Besides the improved convenience, the patch results in depot archives
that are much closer tailored to the actual board by omitting files for
architectures that are not used by the board. E.g., the src/base-hw-pc
archive does not contain any ARM-related content.

The patch also restores the package builds for core/bootstrap object
files as a follow-up fix of "base: remove SPEC variables of boards",
which happened to left 'BOARD' undefined in the src archives.

Fixes #4019
 2021-02-23 12:02:45 +01:00
Josef Söntgen
1b77cb3832 qemu-usb: add ring-buffer for ISOCH OUT transfers 
Be more in line with the original host implementation in Qemu and
buffer transfers. Having transfers in-flight helps to smoothen playback
in case other components utilize the CPU.

Issue #4018.
 2021-02-23 12:02:44 +01:00
Josef Söntgen
19d9409a34 qemu-usb: adapt ISOCH IN handling for audio 
Linux as well as Windows guests want queue to a varying number of
transfers, where each transfer only covers one transaction (iso frame).
The best results were obtained by following that behaviour rather
than queuing multiple iso frames per requests (like is done with OUT
transfers).

The number of requests queued is increased to 32 while the number of
packets per reques is decreased to 1.

Issue #4018.
 2021-02-23 12:02:44 +01:00
Josef Söntgen
9918a8f88d qemu-usb: update port to 5.2.0 
The contrib code is updated from 2.4.1 to version 5.2.0 and the used
device-model is changed to QEMU xHCI. Due to this change older guests
OSes, namely Windows 7, that relied on the NEC xHCI device-model will
not work anymore.

The 'Qemu::Controller' interface was extended by an 'info' method, which
returns the vendor and product ID. This allows for removing the
hard-coded values in the VirtualBox glue code.

Issue #4018.
 2021-02-23 12:02:44 +01:00
Josef Söntgen
a6f0b05834 usb_host: properly handle ISOC read requests 
* Update the 'packet_size' information with the actual length for
  each isoc frame to be able to handle short reads at the client side.

* Copy the whole transfer buffer because the host controller stores
  the data at the original offsets, i.e., the buffer is not densely
  packed.

Fixes #4018.
 2021-02-23 12:02:44 +01:00
Josef Söntgen
b51ae104c2 qemu-usb: use bounce buffer to access DMA memory 
The former implemention assumed that the guest physical memory is
mapped continously. This, however, is not true. Writing larger
files to an USB stick with a Windows 10 guest would therefore lead
to data corruption.

The current implementation uses a bounce buffer to copy the data
to and from the guest physical memory and leaves dealing with the
memory mappings entirely up to the VMM.

Fixes #4017.
 2021-02-23 12:02:44 +01:00
Martin Stein
23620942bf usb_net: allow calling netif_stop_queue dummy 
When the usb_net_drv was introduced in ports/run/netperf.inc, the
netperf_lxip_usb test on x86_64/hw/pc triggered the calling of the
netif_stop_queue dummy at
contrib/<DDE_LINUX>/src/drivers/usb_net/drivers/net/usb/usbnet.c:1464. As
netif_start_queue was also a dummy and allowed to be called, we tried allowing
calls to the netif_stop_queue dummy as well which fixed the
netperf_lxip_usb test on x86_64/hw/pc.

Ref #3961
 2021-02-23 12:02:44 +01:00
Norman Feske
a99f6a81b6 libc: clone 'environ' pointer value after fork 
The 'environ' pointer is a global variable that must be considered as
part of the application state. It must be copied from the parent to the
child process during fork. Otherwise, a child returning from fork is
unable to access environment variables before invoking execve. The
actual environment variables and their values are already captured
because they reside at the application heap, which is cloned from the
parent. So the copied 'environ' pointer refers to valid data.

Fixes #4015
 2021-02-23 12:02:44 +01:00
Sebastian Sumpf
fd0e6685fc riscv: Use Qemu and update to ISA v1.10 
- remove Spike/BBL support in favour of Qemu (>=4.2.1)
- add 'riscv_qemu' board, remove 'spike' board'
- update to privileged ISA v1.10 (from v1.9.1)
- use direct system calls for privileged core threads (they call into
  the kernel and don't use mode changing system calls, i.e. 'ecall',
  semantics)
- use 'OpenSBI' semtantics for SBI calls (to machine mode) instead of
  BBL

issue #4012
 2021-02-23 12:02:44 +01:00
Alexander Boettcher
18e282ab8a allocator_avl: avoid false dangling warnings 
By first removing unused ranges, implicitly meta data allocations are freed
up. This leads to more unused slab blocks and freed up meta data allocations
in the avl tree.

Issue #4014
 2021-02-23 12:02:44 +01:00
Alexander Boettcher
1e84b46c3f slab: add option to free empty blocks 
explictly by a method

Issue #4014
 2021-02-23 12:02:44 +01:00
Martin Stein
19d0142e10 ports/recipes/src: usb_net_drv 
Ref #3961
 2021-02-23 12:02:44 +01:00
Martin Stein
983a18d06e ports: rename *_usb30.run *_usb.run 
Ref #3961
 2021-02-23 12:02:43 +01:00
Christian Prochaska
f654e6f02d usb_host_drv: dwc_otg stability improvements 
- improve NYET error handling
- use split interrupt transaction hack
- clear TT buffers in error situations

Issue #3247
 2021-02-23 12:02:43 +01:00
Sebastian Sumpf
cb2e27f8e4 riscv: Fix link errors for Genode nightly 
- add .got filter rule for trace policies, otherwise ld raises a
  segmentation fault
- force soft-float for binary sources (SRC_BIN)

issue #4012
 2021-02-23 12:02:43 +01:00
Alexander Boettcher
c58acd0b2b allocator_avl: avoid memory leakage 
Fixes #4013
 2021-02-23 12:02:43 +01:00
Alexander Boettcher
26506673c4 seoul-auto: limit to one CPU on Qemu 
Issue #3965
 2021-02-23 12:02:43 +01:00
Norman Feske
df38140ed6 base-pistachio: coding style 2021-02-23 12:02:43 +01:00
Norman Feske
9633a0a524 wm.run: increase RAM quota for wm_backdrop 
Overcomes a startup issue of the wm_backdrop component on base-linux:

[init] child "wm_backdrop" requests resources: ram_quota=10855936
 2021-02-23 12:02:43 +01:00
Norman Feske
7d568247e3 wm: fix deadlock in destroy-view handling 
The deadlock reported in #3236 could be reproduced via the wm.run script
and the modification of test/nitpicker in commit "nitpicker: fix destroy
with invalid handle" by clicking on the testnit entry of the launchpad.

This patch fixes the deadlock by releasing the locked pointer early in
the destruction path, which is legitimate as the wm is single-threaded.

Fixes #3236
 2021-02-23 12:02:43 +01:00
Norman Feske
b5fb37ddee nitpicker: invalidate dangling hovered ptr 
This patch adds the missing invalidation of the _hovered pointer to
User_state::forget, which is required to cover the situation where the
owner of the hovered view vanishes.

Fixes #4011
 2021-02-23 12:02:43 +01:00
Alexander Boettcher
d29b843a0f dde_ipxe: improve throughput for Realtek devices 
By increasing the ring descriptor count the throughput increased from
few kB/s to the range of MB/s.

Issue #3939
 2021-02-23 12:02:43 +01:00
Alexander Boettcher
8958c769ab dde_ipxe: poll once again on irq without rx packets 
This quirk is required for Realtek devices.

Fixes #3939
 2021-02-23 12:02:43 +01:00
Christian Helmuth
210f5073e3 platform_drv.inc: check board=pc not spec=x86 
Fixes run scripts for Linux/x86.
 2021-02-23 12:02:43 +01:00
Martin Stein
ef88d05f2b os: api/net archive 
Ref #3961
 2021-02-23 12:02:43 +01:00
Alexander Boettcher
d6a5a66623 vbox5: enable 64bit SVM support 
Fixes #3965
 2021-02-23 12:02:42 +01:00
Alexander Boettcher
d186e4361e Measure TSC variance of CPUs in test-smp 
Issue #3965
 2021-02-23 12:02:42 +01:00
Emery Hemingway
2acfacb639 genode_dyn.ld: do not emit PHDR segment 
Recent versions of binutils check that PHDR segments are covered
by a LOAD segment. In this case the unloaded PHDR segment can be
ommited.

Fix #4003
 2021-02-23 12:02:42 +01:00
Alexander Boettcher
696d8f030f libgmp: avoid undefined gmpn_*_2 symbols (x86_64) 
__gmpn_addmul_2
__gmpn_mul_2

Fixes #4001
 2021-02-23 12:02:42 +01:00
Alexander Boettcher
e3233a4824 ldso: turn deadlock message into Linker message 
Issue #3991

[init -> ...] Error: LD: jump slot relocation failed for symbol: 'XYZ'
[init -> ...] Error: deadlock ahead, mutex=0x161860, return ip=0xeb870

turns into:

[init -> ...] Error: LD: jump slot relocation failed for symbol: 'XYZ'
[init -> ...] Error: Uncaught exception of type 'Linker::Not_found'
[init -> ...] Warning: abort called - thread: ep

Fixes #4002
 2021-02-23 12:02:42 +01:00
Christian Prochaska
5c5d16f524 ports: add 'vbox5_genode_usb_hid_raw' run script 
Fixes #4009
 2021-02-23 12:02:42 +01:00
Christian Prochaska
c16611dff2 usb_hid_raw.run: add "Timer" service route 
Fixes #4008
 2021-02-23 12:02:42 +01:00
Christian Prochaska
33406940f3 usb_hid_drv: stability improvements 
Fixes #4007
 2021-02-23 12:02:42 +01:00
Christian Prochaska
e1698cf200 dde_linux: fix possible memory leak in wait queue implementation 
Fixes #4006
 2021-02-23 12:02:42 +01:00
Christian Prochaska
2670ae399b usb_host_drv: improve error handling 
Fixes #4005
 2021-02-23 12:02:42 +01:00
Christian Helmuth
91a7fb1da7 cxx: localize operator new(unsigned long) in supc++.o 
Otherwise, base code in ld.lib.so may accidentally use 'new' without
passing an allocator.
 2021-02-23 12:02:41 +01:00
Martin Stein
a9c4ebc9e9 drivers/nic/mode: exception on malformed values 
Ref #3961
 2021-02-23 12:02:41 +01:00
Norman Feske
e3783b00bb coding_style.txt: initializers, control flow 2021-02-23 12:02:41 +01:00
Norman Feske
493924a35e base-hw: coding style 
Improve consistency with the other base repositories, in particular

- Indentation of class initializers
- Vertical whitespace around control-flow statements
- Preferably place control-flow statements (return, break, continue) at
  beginning of a line
- Placing the opening brace of a namespace at the end of line
- Placing the opening brace of a class at a new line
- Removing superfluous braces around single statements
- Two empty lines between methods/functions in implementation files
 2021-02-23 12:02:41 +01:00
Norman Feske
cbae9bc1c8 Add ccache support to build system 
This patch simplifies the use of ccache with the build system. Up until
now, each developer had to set up the ccache hooks manually, adjust the
PATH variable, and customize the etc/tools.conf in each build directory.
With the patch, ccache can be enabled by un-commenting a single line in
the etc/build.conf file.

Fixes #4004
 2021-02-23 12:02:41 +01:00
Norman Feske
8cc2662aac Remove support for the Muen separation kernel 
Fixes #3995
Fixes #3994
 2021-02-23 11:55:44 +01:00
Christian Helmuth
af9ab9190b sd_card_bench.run: add missing braces 
This is a follow-up fix for "base: remove SPEC variables of boards (fix #3971)".
 2021-02-23 11:53:59 +01:00
Christian Helmuth
14db22c77c nic_router.run: disable test 2 UDP 
Issue #3992 is still pending but has low priority, so disable the test
to reduce the noise and increase the chance to spot independent
regressions.
 2021-02-23 11:53:42 +01:00
Martin Stein
691be92046 Don't use the NIC server mode of NIC drivers 
Issue #3961
 2021-02-23 11:53:41 +01:00
Christian Helmuth
9f3c5d92b3 base: cleanup Noncopyable declarations/coding style 2021-02-23 11:51:57 +01:00
Josef Söntgen
36b55e065a usb_host: check ep pointer before using it 
The Usb session allows for submitting packets even when the interface
in question is not yet enabled. Enabling an interface will configure
the udev members properly and is normally done implicitly during
processing of an 'ALT_SETTING' packet.

In case the interface was not enabled this leads to a page-fault in
the USB host-controller driver as 'ep' is NULL.

Fixes #3999.
 2021-02-23 11:51:57 +01:00
Alexander Boettcher
6789ce8b83 platform_drv(x86): support dynamic reconfiguration 
Fixes #3989
 2021-02-23 11:51:39 +01:00
Norman Feske
a981fb864c depot: update recipe hashes 2021-01-25 14:00:43 +01:00
Norman Feske
c4cf9b6e6d base-hw: introduce board-property directories 
This patch moves the bootstrap-link-address information from the
tool/run/boot_dir/hw file to board-specific property files that can be
accessed by using the board as key. This eliminates the need to
customize boot_dir/hw when hosting board-support in an external
repository.

Fixes #3998
 2021-01-25 14:00:43 +01:00
Norman Feske
4bc9b9a2ef tool/run: coding style 2021-01-25 14:00:43 +01:00
Piotr Tworek
ad4211ae2c os: Make Redistributor class inherit Genode::Interface. 
The Vmm::Gic::Gicd_banked::Redistributor is used with Constructible, but
does not have a virtual destructor even though it has virtual methods.
This prompts clang to issue the following warning:

  reconstructible.h:122:4: warning: destructor called on non-final
  'Vmm::Gic::Gicd_banked::Redistributor' that has virtual functions
  but non-virtual destructor [-Wdelete-non-abstract-non-virtual-dtor]

Fix this by inheriting Genode::Interface.

Issue #3984
 2021-01-25 14:00:43 +01:00
Norman Feske
ff28ed0f8c base: avoid superfluous postprocessing of ldso 
Even though the binary patching of ldso must be performed only once,
this postprocessing step was executed on each run because the
postprocess.tag rule is phony (the tag file is never created).
This patch removes this phony behavior by creating the tag file.

Issue #3974
 2021-01-25 14:00:43 +01:00
Tomasz Gajewski
693a4d78dd lib.mk: avoid checking abi on every build 
Fixes #3974
 2021-01-25 14:00:43 +01:00
Piotr Tworek
8f6b934caa base-hw: Fully define _crt0_start_stack value on arm_64 
The _crt0_start_stack label points to a memory location containing the
size of the bootstrap stack. On AArch64 this should be an 8 byte value,
but the code only only defines half of those using asm .long statement.
The other half is expected to be 0, which is true when using GNU as.
This is not the case when using clang's integrated as however. Since
_crt0_stack_size is defined inside .text section clang uses 0xd503201f
value (aarch64 nop instruction) to fill the extra 4 bytes.

Fix this minor incompatibility by explicitly defining both halfs of
this 8 byte quantity.

Fixes #3987
 2021-01-25 14:00:43 +01:00
Piotr Tworek
384cf14bee libc: Remove unused variables from Libc::Kernel. 
The _dispatch_pending_io_signals and _original_suspended_callback member
variables are not used anywhere in the code. This prompts clang to produce
a warning message about it. Remove both variables to fix it.

Issue #3985
 2021-01-25 14:00:42 +01:00
Piotr Tworek
90b20b4daf nitpicker: Fix unsigned long to unsigned int cast. 
The "unsigned Nitpicker::Gui_session::layer()" function returns
~0UL. This prompts clang to produce a warning about implicit unsigned
long to unsigned int conversion. Fix it by returning ~0U instead of
~0UL.

Issue #3985
 2021-01-25 14:00:42 +01:00
Piotr Tworek
80318b9ae0 libc: Add semicolons after [[fallthrough]] 
GCC doesn't care, but clang complains if [[fallthrough]] is not followed
by a semicolon. Existing Genode code is also not consitent in this regard.
This patch adds the extra semicolons since it works in both GCC and
clang.

Issue #3985
 2021-01-25 14:00:42 +01:00
Piotr Tworek
fce5c249c2 gems: Drop unused variables from Window_list class 
No code in this class uses this private member variable. This prompts
clang to produce a warning message about it. Fix it by dropping the
variable.

Issue #3985
 2021-01-25 14:00:42 +01:00
Piotr Tworek
71abfb3b4f libc: Mark Libc::Vfs_plugin class as final. 
The class has final destructor, but is not itself final. This prompts
clang to produce the following warning message:

class with destructor marked 'final' cannot be inherited from [-Wfinal-dtor-non-final-class]
                ~Vfs_plugin() final { }
                              ^
vfs_plugin.h:39:13: note: mark 'Libc::Vfs_plugin' as 'final' to silence this warning

Issue #3985
 2021-01-25 14:00:42 +01:00
Piotr Tworek
395a9b5bf5 libc: Remove unused print function from select.cc. 
This static inline function is not used anywhere. GCC does not care,
but clang warns about this. Remove the function to allow the code to
compile cleanly with both clang and GCC.

Issue #3985
 2021-01-25 14:00:42 +01:00
Piotr Tworek
53081ac6b3 libc: Dont't use local_addr in try/catch block. 
The code in Libc::Cloned_malloc_heap_range initializes its local_addr
member variable by calling Region_map::attach_at. This function can
throw Region_conflict exception. The handler for this exception uses the
local_addr to produce an error message. Such error log is IHMO
useless, or even incorrect since the value of local_addr is undefined
in such case.

Its also worth noting that clang 12 produces the following warning for
this code:
  "cannot refer to a non-static member from the handler of a constructor
   function try block"

Issue #3985
 2021-01-25 14:00:42 +01:00
Piotr Tworek
c6d5b98227 decorator: Remove unused _topped_cnt variable. 
This private member variable is not used anywhere in the code. This
produces compilation warning when using clang instead of GCC. Drop the
unused variable.

Issue #3985
 2021-01-25 14:00:42 +01:00
Piotr Tworek
c402cc1045 window_layouter: Drop unused code. 
The State enum and _state private member variable are not used anywhere.
This produces an extra warning when building the code with clang instead
of GCC. Remove this dead code.

Issue #3985
 2021-01-25 14:00:42 +01:00
Piotr Tworek
1edac9730c scout: Instantiate templates in their namespace. 
When building the code with clang the following warning message is
prodiced:

  "explicit instantiation of 'Scout::Browser_window' must occur in namespace 'Scout'
   template class Browser_window<Genode::Pixel_rgb888>"

 This happens for several different types. This patch fixes the problem
 by instantiating all those templates using their explicit full name.

Issue #3985
 2021-01-25 14:00:14 +01:00
Piotr Tworek
d475015ada launchpad: Fix _spacer member initialization. 
The Scout::Spacer constructor requires two arguments. The initialization
of the type in Launchpad_window declaration does not specify them. The
variable is however initialized a second time in class constructor. This is
most likely why GCC accepts this code. Clang on the other hand
complains about it.

Fix this by properly initializing both _spacer and _docview only once at
declaration time.

Issue #3985
 2021-01-25 13:58:11 +01:00
Piotr Tworek
ffb931f8b1 nano3d: Fix header guard in sqrt.h 
The code fails to build with clang due to the following warning/error:

  error: '_INCLUDE__NANO3D__SQRT_H_' is used as a header guard here,
  followed by #define of a different macro [-Werror,-Wheader-guard]

Fix this by removing the extra underscore from this header guard
definition.

Issue #3985
 2021-01-25 13:58:11 +01:00
Piotr Tworek
b4d294f62e os: Drop unused Net::Arp_cache::_init member. 
Not used anywhere in the code. Clang 11 complains about this.

Issue #3984
 2021-01-25 13:58:11 +01:00
Piotr Tworek
052f678225 os: Add missing override keyword. 
The _device_specific_features() implements part of the Virtio_device
interface. Decorate it with override keyword to make this clear and
also satisfy clang which produces warning regarding this.

Issue #3984
 2021-01-25 13:58:11 +01:00
Piotr Tworek
3fdf323e6e os: Drop unused Net::Dhcp_client::_alloc member. 
Not used for anything so technically a dead code. Clang 11 complains
about this.

Issue #3984
 2021-01-25 13:58:11 +01:00
Piotr Tworek
05c36d67ce os: Fix clang constant warning in Port_allocator. 
Clang 11 produces the following warning when building port_allocator.cc:

port_allocator.cc:27:21: error: result of comparison of constant 65536 with
expression of type 'const Genode::uint16_t' (aka 'const unsigned short') is
always true [-Werror,-Wtautological-constant-out-of-range-compare]
               (port.value <  (unsigned)(Port_allocator::FIRST +

Basically the code compares Port::value (uint16_t) against a constant
65536 which is larger than UINT16_MAX (65535). This comparison will always
be true.

Issue #3984
 2021-01-25 13:58:11 +01:00
Piotr Tworek
ffc2a2f306 os: Disambiguate File_system in ram_file_system.h. 
According to clang there are two symbols which File_system can refer to:
1. namespace File_system from ram_fs/chunk.h.
2. Vfs::File_system class from  vfs/file_system.h.
Make it clear we refer to the File_system namespace in this case.

Issue #3984
 2021-01-25 13:58:11 +01:00
Piotr Tworek
fc089a1673 nic_dump: Remove unused Net::Interface::_alloc 
This private member variable is not used anywhere in the code. This
breaks builds using clang instead of GCC. Drop the variable.

Issue #3984
 2021-01-25 13:58:11 +01:00
Piotr Tworek
428de89f9a nic_dump: Disambiguate "Interface" in component.cc. 
Clang 12 complains that Interface can both refer to Genode::Interface
and Net::Interface in this case. Explicitly state the code refers to
the latter.

Issue #3984
 2021-01-25 13:58:11 +01:00
Piotr Tworek
30429a5228 nic_flood: Drop Dhcp_client::_alloc member. 
This private member variable is not used anywhere in the code. This
breaks builds using clang instead of GCC. Drop it.

Issue #3984
 2021-01-25 13:58:11 +01:00
Piotr Tworek
e6a9e06f62 base-linux: Drop parentheses from .cfi_undefined. 
According to GNU as manual the syntax of this directive is:
  .cfi_undefined register

The manual does not mention the register should be in parentheses.
This works in GNU as even when those are present, but unfortunately
clang integrated-as does not parse this correctly. Both GNU and
clang's integrated assembler work fine when the extra parentheses
are omitted.

Fixes #3986
 2021-01-25 13:58:10 +01:00
Piotr Tworek
8b172bf22e base: Explicitly state ELF segment flags 
The code in base-hw/src/bootstrap/platform.cc uses segment flags for
identification purposes. Based on this information the code decides
what to do with each segment. Unfortunately the linker script does
not actually ensure the flags for a specific named segment match
expectations. The code relies on implicit linker behaviour.
This implicit behaviour can vary between linkers. This breaks
arm_v7a base-hw builds linked with LLVM's lld linker. The segment
named "ro" ends up having writeable flag set when using LLD.

This patch ensures that all ELF segments in genode.ld have their
required perimssion flags set explicitly.

Fixes #3988
 2021-01-25 13:58:10 +01:00
Piotr Tworek
80e8cf99e2 base: Make Genode::List clang friendly. 
Clang is generally fine with Genode::List and compiles code using it
without emitting any warnings. There is however one exception. Clang
fails hard when building base-hw/src/core/kernel/object.cc.
This is due to a call to Genode::List::remove made from
Object_identity::invalidate function. The error message clang
produces is:

  list.h:96:33: error: 'Genode::List<Kernel::Object_identity_reference>::Element::_next'
  is not a member of class 'const Kernel::Object_identity'
                              _first = le->List::Element::_next;
                                           ~~~~~~~~~~~~~~~^

When we look at the declaration of the Kernel::Object class on which
the remove method is called. as expected it does inherit Genode::List:

using Object_identity_list
	= Genode::List<Kernel::Object_identity>;

class Kernel::Object : private Object_identity_list
{
...
}

Given the error message we see that List::Element should be resolved to
Genode::List<Kernel::Object_identity>::Element, and not
Genode::List<Kernel::Object_identity_reference>::Element. But how does
clang manage to figure out we're talking about Object_identity_refecence
list here? Well, I admit I don't know the exact steps it takes to arrive
at this conclusion, but it is not entirely wrong. If we take a look at
what Kernel::Object_identity is we'll see:

class Kernel::Object_identity
: public Object_identity_list::Element,
  public Kernel::Object_identity_reference_list
{
...
}

Where as one can guess Object_identity_reference_list is defined as:

using Object_identity_reference_list
	= Genode::List<Object_identity_reference>;

Long story short Kernel::Object has Genode::List of both Kernel::Object_identity
and Kernel::Object_identity_reference in its inheritance chain and clang
is not really sure to which of those the code refers to in
Genode::List::remove method by using List::Element::.

The fix for this is relatively simple, explicitly state the full type of
the base class the code intends to refer to. Replacing List::Element,
with List<LT>::Element makes the code buildable with both clang and GCC.

Fixes #3990
 2021-01-25 13:58:10 +01:00
Alexander Boettcher
9d239957bc libc: update seek state on opening O_APPEND fds 
If a fd is opened in append mode and just is to be used (so never written by
the parent component) for the to be forked child, the seek state was not
pointing to the end of the file.  The wrong seek value then was used in
File_descriptor_allocator::generate_info().

Issue #3991
 2021-01-25 13:58:10 +01:00
Alexander Boettcher
5fa91c573b libc: update O_APPEND fds on joined child 
The fds in O_APPEND state may have changed by the child. The local seek
state needs to be valid/current for the next to be forked child, which seek
state is set by File_descriptor_allocator::generate_info().

Issue #3991
 2021-01-25 13:58:10 +01:00
Norman Feske
9bfd812a88 tool/run: search repos for run-tool plugins 
This patch allows for supplementing the existing run-tool plugins with
additional plugins hosted inside a repository. E.g., it enables the run
tool to find world/tool/run/boot_dir/hw automatically.

Fixes #3993
 2021-01-25 13:58:10 +01:00
Norman Feske
1ccf8a280c base-hw: simplify board support in external repo 
Avoid use of REP_DIR in *.mk files to simplify the use of these files as
templates for a board hosted in a separate repository.

Use REP_INC_DIR for searching headers, thereby considering headers
hosted in an external repository.

Issue #3168
 2021-01-25 13:58:10 +01:00
Stefan Kalkowski
f034f560be create_builddir: remove deprecated platforms 
Fix #3996
 2021-01-25 13:58:10 +01:00
Martin Stein
f45aa85e9f nic_router.run: fix redundant IPv4 subnets 2021-01-25 13:58:10 +01:00
Norman Feske
84443d6548 os: div_zero trace policy 
This trace policy can be used as a fault-injection mechanism.
Once installed, the next time the traced thread passes a trace point
(e.g., RPC call, RPC request, ...), the policy produces a divide-by-zero
exception.
 2021-01-25 13:58:10 +01:00
Norman Feske
a6a923c31b driver manager: restart intel_fb when frozen 
This patch introduces heartbeat monitoring and automated restarting
of the intel framebuffer driver.
 2021-01-25 13:58:10 +01:00
Norman Feske
f687d4824b sculpt: set lower bound for font size 2021-01-25 13:58:10 +01:00
Norman Feske
0a478dac7f pdf_view: tweaks for showing 16:10 slides 
- Shave off two pixels on right, which are rendered white for no reason.
- Adjust resolution to make the slides fit vertically by default.
 2021-01-25 13:58:10 +01:00
Norman Feske
5905e0a4a0 Road map for 2021 2021-01-25 13:58:09 +01:00
Martin Stein
d0ac8a6036 cbe_tester.run: use /tmp for image on autopilot 2021-01-25 13:58:09 +01:00
Stefan Kalkowski
bdd923406f base: remove SPEC variables of boards (fix #3971) 
* Remove SPEC declarations from mk/spec
* Remove all board-specific REQUIRE declaratiions left
* Replace [have_spec <board>] run-script declarations with have_board where necessary
* Remove addition of BOARD variable to SPECS in toplevel Makefile
* Move board-specific directories in base-hw out of specs
 2021-01-25 13:58:09 +01:00
Stefan Kalkowski
5a123e37c9 run: introduce have_board helper function 
The new helper function returns a boolean value analoque to 'have_spec' if the
BOARD variable corresponds to the given value. It shall replace [have_spec <board>]
declarations in run-scripts.

Ref #3971
 2021-01-25 13:58:09 +01:00
Christian Helmuth
6cfaac182a Remove Cpu_session::Native_cpu definition from API 
This type can be a forward declaration in the public API because its
definition is required only in kernel-specific code.

Related to #3979
 2021-01-25 13:58:09 +01:00
Christian Helmuth
3e73d8d7b6 drivers_nic-muen: permit Timer access in platform_drv 2021-01-25 13:58:09 +01:00
Pirmin Duss
a4d5687510 base-hw: allow out of tree imx8 platforms 
Issue #3911
 2021-01-25 13:58:09 +01:00
Alexander Boettcher
2b0170fb6a base-hw: ack thread capability on construction 
Fixes #3982
 2021-01-25 13:58:09 +01:00
Martin Stein
2d21d04c76 cbe_tester.run: raise timeout to 6 minutes 2021-01-25 13:58:09 +01:00
Martin Stein
f6d195a9de nic drivers: provide optional Uplink-client mode 
In order to perform a smooth transition from NIC drivers that act only as NIC
session clients to NIC drivers that act only as Uplink session clients, this
commit introduces an intermediate state in which all NIC drivers support both
modes. That said, a NIC drivers mode is now statically determined through a new
optional 'mode' attribute in the drivers <config> tag that can be set to either
'nic_server' (default value) or 'uplink_client'. Reconfiguring this attribute
at a driver doesn't have any effects. Whithout this attribute being set, all
NIC drivers will behave the same as they did before the commit. When set to
'uplink_client', however, instead of providing a Nic service, they request
an Uplink session whenever their network interface becomes "UP" and close the
session whenever their network interface becomes "DOWN".

Ref #3961
 2021-01-25 13:58:09 +01:00
Martin Stein
1d2649b49a nic_router: act as "Uplink" server 
Let the NIC router provide an Uplink service besides the Nic service that it
already provided. Requests for an Uplink session towards the NIC router are
assigned to Domains using the same <policy> configuration tags that are used in
order to assign Nic session requests. The MAC addresses of Uplink session
components are _NOT_ considered during the allocation of MAC addresses for NIC
session components at the same Domain. The task of avoiding MAC address clashes
between Uplink session components and Nic session components is therefore left
to the integrator. Apart from that, Uplink session components are treated by
the NIC router like any other interface.

Ref #3961
 2021-01-25 13:57:43 +01:00
Martin Stein
cf72d1aac3 os: introduce new "Uplink" session 
Adds new Uplink session interface, the corresponding client side (Client,
Connection), and the corresponding API archives. An Uplink session is almost
the same as a NIC session with the difference that the roles of the end points
are swapped. An Uplink client is the one that provides a network interface
(for instance, a NIC driver) whereas an Uplink server is the one that uses
that network interface (for instance, a networking stack).

Therefore, in contrast to the NIC session, MAC address and link state come from
the Uplink client. The link state is reflected through the lifetime of an
Uplink session: The client requests the session only when the link state is
"UP" and closes it whenever the link state becomes "DOWN" again. The MAC
address is transmitted from the Uplink client to the Uplink server as an
argument of the session request.

Ref #3961
 2021-01-25 13:48:09 +01:00
Martin Stein
9222463565 run/ping: support running manually on Linux 
* Adds documentation how to prepare and finalize a Linux for running the
  scenario ontop of it
* Adds consideration of env variable 'ON_LINUX_WITH_DST_IP' that, if set,
  adapts the run script to running on Linux with the given ping destination IP

Ref #3961
 2021-01-25 13:48:09 +01:00
Norman Feske
8ff75346dd sticks_blue_backdrop: inherit roms from backdrop 
This patch moves the <content> <rom/> ... </content> declarations that
refer to pkg/backdrop to the runtime file of pkg/backdrop.

Issue #3981
 2021-01-25 13:48:09 +01:00
Norman Feske
cae5d380c4 depot_query: inherit pkg rom content from deps 
This patch relaxes the need to specify the rom content of all pkg dependencies
in each runtime file. Whenever a dependency features a runtime file, the
contained <content> <rom/> ... </content> nodes are implicitely included.

Fixes #3981
 2021-01-25 13:48:09 +01:00
Alexander Boettcher
14d8627186 platform_drv: avoid using Register in Bdf type 
Issue #3963
 2021-01-25 13:48:09 +01:00
Alexander Boettcher
f358fcbda6 nova: flush Intel IOMMU caches 
- when memory is detached

Issue #3963
 2021-01-25 13:48:09 +01:00
Alexander Boettcher
b185f3fac1 platform_drv: add support to power PCI devices 
- depending on available PCI power cap power off and on
- save and restore PCI bars if required
- reset PCI devices after power on if supported

Fixes #3963
 2021-01-25 13:48:09 +01:00
Alexander Boettcher
5f7fe7498f platform_drv: add mmio delayer support 
required after power on and function level reset

Issue #3963
 2021-01-25 13:48:08 +01:00
Alexander Boettcher
c89864c830 platform_drv: add typesafe PCI config access 
using Register_set && Register. It is derived from Genode::Mmio, however uses
as backend the Config_access implementation which is still used widely in
the platform driver.

Issue #3963
 2021-01-25 12:51:54 +01:00
Alexander Boettcher
59fafac4d6 platform_drv: increase readability 
by adding convenience functions to make code easier readable

Issue #3963
 2021-01-25 12:51:54 +01:00
Alexander Boettcher
ebf7f8f599 platform_drv: introduce structured PCI BDF type 
Replace explicit usage of bus, device, function arguments to methods or
variables all over the code by a single data type. It eases the reading of and
shorten the code.

Issue #3963
 2021-01-25 12:51:54 +01:00
Norman Feske
f57519397b Remove Pd_session::Native_pd definition from API 
This type can be a forward declaration in the public API because its
definition is required only in kernel-specific code.

Fixes #3979
 2021-01-25 12:51:54 +01:00
Christian Helmuth
5ca3847c89 cbe_tester requires x86_64 currently 2021-01-25 12:51:53 +01:00
Norman Feske
eee8f64fd4 sculpt: fix dialog sizes with manual fonts config 
Fixes #3976
 2021-01-25 12:51:53 +01:00
Alexander Boettcher
0a5741f076 acpica.run: add acpi_hid evaluation support 
Evaluate acpi_hid report in acpi_event

Showcase feature in acpica.run for FUJITSU FUJ02E3 and KEY_FN_F4.
 2021-01-25 12:51:53 +01:00
Alexander Boettcher
1147f35972 acpica: add limited support for FUJ02E3 ACPI dev 
The keys, mute, touchpad toggle (Fn-F4) and rfkill may be reported by some
Fujitsu machines via the ACPI FUJ02E3 ACPI device. With this commit limited
support to detect the 3 keys are added and will be reported as Genode report.
 2021-01-25 12:50:55 +01:00
Norman Feske
d698e0876d window_layouter/decorator: drag-state corner cases 
This patch complements the commit "nitpicker: defer hover changed while
dragging" with fixes of the window layouter and motif decorator.

- Handling of empty pointer model (after unhovering a decoration)
  instead of not updating it.

- Re-applying the hover model to the window layout when leaving the
  drag state. This addresses the corner case that the hover model
  changed during the drag operation (which is rightfully not handled
  while dragging).

- Letting the window layouter enter the drag state only if a dragged
  window is defined.

Issue #3973
 2021-01-25 12:50:55 +01:00
Tomasz Gajewski
98798f18b5 Fix minor inconsistencies in mk files 
Fixes #3972
 2021-01-25 12:50:54 +01:00
Christian Helmuth
8bed4c1d54 base: support hexa-decimal values in ascii_to(long) 
This aligns the behavior of ascii_to(long&) with the behavior of
unsigned integer types, e.g., when parsing XML nodes into signed long
variables.
 2021-01-25 12:50:54 +01:00
Norman Feske
72801975cd nitpicker: defer hover changed while dragging 
When holding at least one key or button, the hovering should never
change. In the previous version, this invariant was not enforced,
but the symptoms for eventual inconsistencies were masked by two
checkes: one when generating the hover report, and one when submitting
the leave event. This patch enforces the invariant by suppressing any
change of 'User_state::_hovered' while in dragged state.

Fixes #3973
 2021-01-25 12:50:54 +01:00
Norman Feske
7266f29491 decorator: disable sync handling when idle 
When idle, the decorator still consumes a bit of CPU time when triggered
by the GUI session's sync signal. This patch installs the sync handler
each time the decorator goes idle.
 2021-01-25 12:50:38 +01:00
Norman Feske
2c82636a98 terminal: respond to font-size changes 
Issue #3970
 2020-12-23 13:38:39 +01:00
Norman Feske
d47f87a768 terminal: use GUI session directly 
Issue #3196
Fixes #3970
 2020-12-23 13:38:31 +01:00
Norman Feske
887fcecf63 base-linux: disable initramfs build on 32-bit 
The build tries to access the host header asm/unistd.h, which is
apparently not present on 64-bit hosts (in particular our CI).
 2020-12-18 09:11:25 +01:00
Christian Prochaska
0428e5e8b9 qt5: improve quickcontrols2 module support 
Fixes #3968
 2020-12-18 09:11:25 +01:00
Alexander Boettcher
0359ee6a76 sandbox: fix position scaling in affinity filter 
During position scaling use the location width/height which is the permitted
dimension within the child space, instead of using the whole possible
child space. (Fixup for "init: scale affinity location" Issue #1412)

Thanks to the report by Peter.
 2020-12-18 09:11:25 +01:00
Christian Helmuth
1bef11accf depot: update recipe hashes 2020-12-18 09:10:52 +01:00
Alexander Boettcher
c5de2acf57 vm_session(x86): support cstar register 
Fixes #3964
 2020-12-09 14:02:12 +01:00
Norman Feske
9189342b77 base-okl4: coding style 2020-12-09 14:02:11 +01:00
Norman Feske
abd688097a base-fiasco: coding style 2020-12-09 14:02:11 +01:00
Norman Feske
6930372d55 base-foc: remove stale README content 2020-12-09 14:02:11 +01:00
Martin Stein
a124f5b88d nic_router: support and show-case disabling arp at a domain 
* Introduce 'use_arp' configuration flag at the NIC router in order to disable
  requesting IP-to-MAC address resolutions via ARP for domains.
* Add automated run/nic_router_disable_arp test to the autopilot that tests
  the effect of the new flag
* Document the new flag and the new test in the NIC router README

Fixes #3935
 2020-12-09 14:02:11 +01:00
Sebastian Sumpf
0beda6bca4 dde_linux: USB MBIM modem driver support 
Thise driver supports USB LTE modems for Huawais' ME906s through MBIM
and provides a traditional Nic session. The "control" interface is a
Terminal session, which can be used via libmbim/mbimcli.

issue #3822
 2020-12-09 14:02:11 +01:00
Norman Feske
a0fb944721 Propagate session diag flag to core 
This commit restores the diag feature for selecting diagnostic output of
services provided by core. This feature became unavailable with commit
"base: remove dependency from deprecated APIs", which hard-wired the
diag flag for core services to false.

To control this feature, three possible policies can be expressed in a
routing target of init's configuration:

* Forcing silence by specifying 'diag="no"'
* Enabling diagnostics by specifying 'diag="yes"'
* Forwarding the preference of the client by omitting the 'diag'
  attribute

Fixes #3962
 2020-12-09 14:02:11 +01:00
Sebastian Sumpf
36eeab6df2 dde_linux: usb support extra data and zero config 
- Vendor devices add addtional data to the config descriptor, read and
  added to the usb session
- allow '0' configuration within the usb session

issue #3822
 2020-12-09 14:02:11 +01:00
Sebastian Sumpf
537472e9af libports: ffi for ARM_64 
Required by 'glib'

issue #3822
 2020-12-09 14:02:10 +01:00
Christian Helmuth
496dc5508f usb_hid: support for Apple HID devices 
Issue #3953
 2020-12-09 14:02:10 +01:00
Christian Helmuth
2a659cb750 usb_hid: support for Holtek-based mice fixups 
This enables USB mice based on the Holtek chipsets, which need USB
descriptor fixups. An example mouse is the Sharkoon Drakonia.

Issue #3953
 2020-12-09 14:02:10 +01:00
Alexander Boettcher
b097e598f1 usb_hid: use heap instead of dma/mem allocator 
The usb_hid driver does not need to distinguish between
normal memory and DMA capable memory, since all requests are routed via the
USB raw session to the usb host driver. The default Malloc implementation
implements this distinction, however exposes restrictions on the size of
allocations. As seen now by several USB HID devices, the size of device
driver allocations depend on read out hardware features and can be
larger than we support with our specialized default Malloc implementation.

Since we don't need this functionality, switching to an well
tested allocator (Heap) which can cope with varying sizes of allocation,
we can mitigate the size restriction.

Fixes #3953
 2020-12-09 14:02:10 +01:00
Alexander Boettcher
2c639169fd dde_linux: use linux memory primitives in impl/usb 
Issue #3953
 2020-12-09 14:02:10 +01:00
Martin Stein
bad8caee3f nic_router: support multiple DHCP option 6 entries 
* The NIC router now considers, memorizes, and, if configured, reports
  multiple DHCP option 6 entries from DHCP replies that it received as DHCP
  client
* A DHCP server at the NIC router can now be configured statically with
  multiple DNS server addresses to propagate
* The 'dns_server_from' attribute of the DHCP server of the NIC router now
  supports the forwarding of multiple DNS server addresses
* The automated run/nic_router_dhcp test tests all the above mentioned new
  functionality and reconfiguring it at runtime. The test was added to the
  autopilot.
* All run scripts were adapted to fit the new NIC router configuration
  interface

Fixes #3952
 2020-12-09 14:02:10 +01:00
Emery Hemingway
306466fc60 libc: always set argv and envp to valid arrays 
The arrays passed to main(…) must always be valid null-terminated arrays.

Fix #3955
 2020-12-09 14:02:10 +01:00
Martin Stein
063e4bd072 remove remnants of 'virtualbox_nic_router' test 
Ref #3952
 2020-12-09 14:02:10 +01:00
Martin Stein
e14b58a82c net/dhcp.h: more specific include guards 
Ref #3952
 2020-12-09 14:02:10 +01:00
Martin Stein
8d8edaea5d nic_router: fix link state on dhcp server reconfig 
When reconfiguring the NIC router modifies the information that DHCP clients
received through a DHCP server of the NIC router, the link state of the
correspodning interface has to do a "down-up" sequence. This provides a signal
to the DHCP clients to redo DHCP, receive the new information and update
accordingly.

Fixes #3931
 2020-12-09 14:02:10 +01:00
Martin Stein
b0327d0544 nic_router: preserve session link state until read 
See the NIC router README paragraph "Behavior regarding the NIC-session link
state" for further information.

Ref #3931
 2020-12-09 14:01:42 +01:00
Martin Stein
a7b878cbb5 nic_router: allow for config values of 0 seconds 
Ref #3931
 2020-12-09 14:01:42 +01:00
Martin Stein
7ac6f93838 nic_router: clean up init of configuration members 
Ref #3931
 2020-12-09 14:01:42 +01:00
Norman Feske
70ff3d9c90 base-foc: coding style 
- Cosmetic adjustments according to
  https://genode.org/documentation/developer-resources/coding_style

- Replace manual inclusion of kernel headers by one new compound
  header foc/syscall.h

- Rename namespace Fiasco to Foc
 2020-12-09 14:01:42 +01:00
Norman Feske
0209a2465d base-foc: update cap_integrity.run 2020-11-28 19:37:15 +01:00
Alexander Boettcher
b6408cec1c ssh_terminal: close terminal fd in pthread context 
Issue #3682
 2020-11-27 14:40:41 +01:00
Christian Helmuth
3fac8b106d version: 20.11 2020-11-27 14:01:47 +01:00
Norman Feske
5c27270b17 News item for version 20.11 2020-11-27 13:57:55 +01:00
Norman Feske
3f15d18392 Release notes for version 20.11 2020-11-27 13:57:54 +01:00
Christian Helmuth
f2e0c164c2 depot: update recipe hashes 2020-11-27 09:21:06 +01:00
Norman Feske
d672e95090 doc: update components.txt 2020-11-27 09:19:09 +01:00
Norman Feske
98211db63d doc: move release notes to sub directory 
This keeps the doc/ directory tidy and neat.
 2020-11-27 09:19:09 +01:00
Sebastian Sumpf
722254f864 dde_rump: Set '-fcommon' for rump host tools 
GCC 10 has `-fno-common` as default which leads to multiple definitions of
of symbols.

Fixes #3960
 2020-11-27 09:19:09 +01:00
Stefan Kalkowski
b907629341 sculpt: add basic support for i.MX8 Quad EVK 
* Introduce CPU quota for driver subsytem (needed by sd_card_drv)
* Introduce CPU quota for runtime subsytem and nic_drv (needed by fec_nic_drv)
* Increase CAP quota for inspect terminal slightly
* Add sculpt packages for imx8q_evk

Fix #3958
 2020-11-27 09:19:09 +01:00
Stefan Kalkowski
22852f2e50 sculpt: move pc-specifics away from generic part 
* Introduce sculpt-[board] specific package
* Move rtc driver into managed drivers sub-system
* Name nic_drv in a generic fashion in sculpt_manager
* Copy over pc-specific config files only when building for PC

Ref #3958
 2020-11-27 09:19:09 +01:00
Christian Prochaska
e22e2540ee qt5: add Qt5Xml library 
Fixes #3959
 2020-11-27 09:19:09 +01:00
Alexander Boettcher
78ab3c8db5 usb: avoid memset on failed memory allocation 
Issue #3953
 2020-11-27 09:19:09 +01:00
Alexander Boettcher
ffdd49f9ce platform_drv: free ever non PCI devices on close 
Fixes #3957
 2020-11-27 09:19:09 +01:00
Stefan Kalkowski
0cbd1d1b7c os: adjust sd_card_drv to use platform API 
Fix #3956
 2020-11-27 09:19:09 +01:00
Stefan Kalkowski
f4ac642f64 platform_drv: reset pointer when releasing device 
Ref #3956
 2020-11-27 09:19:09 +01:00
Martin Stein
955afd8837 ada runtime: support image of enums 
Ref #3937
 2020-11-27 09:19:09 +01:00
Norman Feske
9b544787bd base: make Affinity::Space::location_of_index const 2020-11-27 09:19:09 +01:00
Alexander Boettcher
774b1f4277 base: remove Thread_deprecated 
Fixes #3954
 2020-11-27 09:19:08 +01:00
Josef Söntgen
dbcb1ff480 ssh_terminal: flush before closing connection 
Originally the outgoing buffer was flush when the terminal was
detached. Since we cannot do that at this point, make the flushing
part of the common cleanup procedure.

Issue #3682.
 2020-11-27 09:19:08 +01:00
Josef Söntgen
551b17591c ssh_terminal: handle detached term in event loop 
Rather than calling 'ssh_disconnect' from within the Terminal session,
flag the detached terminal in the session and let the event loop do the
cleanup. Otherwise it might happen that the 'ep' (handling the Terminal
session) as well as the 'pthread.0' (executing the ssh event loop) end
up both triggering the cleanup concurrently.

Issue #3682.
 2020-11-27 09:19:08 +01:00
Josef Söntgen
51a50ece60 libc: guard select call in socket_fs with monitor 
Issue #3682.
 2020-11-27 09:19:08 +01:00
Norman Feske
0dcb526ae5 base-linux: coding style 2020-11-27 09:19:08 +01:00
Christian Helmuth
dc016cbd5c ldso: log linker-area info on 'ld_verbose' 2020-11-27 09:19:08 +01:00
Christian Prochaska
e5f442f2d3 qt5: use integrated virtual keyboard in example application 
Fixes #3949
 2020-11-27 09:19:08 +01:00
Piotr Tworek
5db2971903 base-hw/arm_v7a: Mark r1 as out in Kernel::call64 
In case of arm_v7a Kernel::call64 the syscall will use both r0 and r1 as
output registers. Unfortunately the inline asm does not reflect this and
only r0 is explicitly specified as output. GCC manages to produce output
which we'd like to see. Clang on the other hand takes adventage of the fact
r1 should not be touched by swi and produces fewer instructions which do
what the code describes, but not what we actually want.

Basically the code which we want and is generated by GCC is:
  svc     0
  mov     r2, r0
  mov     r0, r1
  mov     r1, r2
  bx      lr

Clang on the other hand generates correct assembly given the code, but
incorrect given what the function is supposed to do:
  svc     0
  mov     r1, r0
  mov     r0, #0
  bx      lr

Both GCC and clang generate the same, expected assembly output when r1
is marked as output register from the inline asm swi call.

Fixes #3951
 2020-11-27 09:19:08 +01:00
Alexander Boettcher
aa7f5bc95f os: add cpu balancer component 
Issue #3843
 2020-11-27 09:19:08 +01:00
Alexander Boettcher
6872fdb0de cpu_burner: utilize all assigned CPUs 
Issue #3843
 2020-11-27 09:19:08 +01:00
Piotr Tworek
48220dfd9b os: Add missing {} around sub-object initializers. 
Clang complains when a sub-object is initialized without {}.

Issue #3950
 2020-11-27 09:19:08 +01:00
Piotr Tworek
50ab86cd72 base-hw: Remove unused rdtsc function for x86_64 
This function has no callers. Clang warns about this.

Issue #3950
 2020-11-27 09:19:08 +01:00
Piotr Tworek
cc7de65c9e nitpicker: Remove unused hover_reported variables 
Fix for unused private member variable warning reported by clang.

Issue #3950
 2020-11-27 09:19:08 +01:00
Piotr Tworek
cc193a9155 base: drop Trace::Session_component::_parent_levels 
Clang correctly asserts this private member variable is not used
anywhere in the code. I'm not sure what the intention of the code is,
might be this is a part of some unfinished feature. This patch just does
the minimum amount of work to allow the code to build with clang. If
required I can also drop the parent_levels constructor argument and
clean up the call sites.

Issue #3950
 2020-11-27 09:19:08 +01:00
Piotr Tworek
c0309a634e base: Silence unused arg warning in rpc_server.h 
The msg argument in Genode::Rpc_dispatcher::_read_arg is not used. GCC
does not care about this, but clang does and prints a warning regaring
this. Silence it by removing unused argument name.

fixup! base: Silence unused arg warning in rpc_server.h
 2020-11-27 09:19:08 +01:00
Martin Stein
30b8f4efc8 gems: import Genode-specific code of the CBE 
The CBE repository contained a lot of Genode-specific code despite the fact
that the CBE core logic is not bound to Genode in any way. Therefore the
Genode-specific CBE code is moved to the 'gems' repository to form part of
Genode mainline. The remaining CBE code becomes a port in Genode instead of
being invoked as sub-repository.

The commit combines the following work steps:

* add all files removed from CBE repository
* add CBE port files
* make all CBE libs and targets build again
* make all CBE run scripts succeed again
* make all CBE recipes build again
* make CBE autopilot succeed again
* let CBE autopilot use 'libsparcrypto' contrib dir and Genode build dir
  instead of '.ci' dir in CBE contrib dir (remove '.ci' dir from CBE repo)
* let CBE autopilot always check for all ports
* make CBE autopilot directly executable
* fix license headers in all Genode CBE files
* remove unused VFS replay component
* remove unused CBE test
* remove unused external crypto
* remove unused files in run dir
* remove unused external trust anchor
* add cbe_tester test to autopilot list
* get rid of directories 'include/cbe_*' and 'include/utils'

Fixes #3937
 2020-11-27 09:19:08 +01:00
Alexander Boettcher
24181f2bf6 driver_manager: use componentized usb driver 
Fixes #3943
 2020-11-23 12:03:00 +01:00
Alexander Boettcher
fae3c12366 usb: skip reporting of incomplete init. usb device 
Fixes #3944
 2020-11-23 12:03:00 +01:00
Stefan Kalkowski
4e90dc4512 dde_linux: adjust fec_nic_drv to use platform_drv 
Fix #3947
 2020-11-23 12:03:00 +01:00
Alexander Boettcher
a4c7837fb3 Update to bender with optional microcode and HWP 
plugin support.

Fixes #3871
 2020-11-23 12:03:00 +01:00
Christian Prochaska
764ab3be20 qt5: use libc 'posix_memalign()' in qtwebengine 
Fixes #3916
 2020-11-23 12:03:00 +01:00
Alexander Boettcher
c6a2e287d0 trace: forward exceptions during construction 
The control area is constructed during session creation and the caller can
handle the Out_of_* exception by increasing the quota by the next attempt.

Fixes #3917
 2020-11-23 12:03:00 +01:00
Emery Hemingway
d16a1bd922 vfs_lwip: log warning when blocking application 
Log a warning when the lwIP VFS plugin queues a read while waiting
for an interface to become ready. By comparision the Lxip plugin
has the same behavior but a more verbose initialization process.

Fix #3920
 2020-11-23 12:03:00 +01:00
Alexander Boettcher
b7ba508110 nova: update to kernel supporting AMD IO-MMU 
Fixes #3928
 2020-11-23 12:03:00 +01:00
Alexander Boettcher
d9cde328cb acpi: support amd iommu detection 
Issue #3928
 2020-11-23 12:03:00 +01:00
Alexander Boettcher
6b20a6bc7c tool: add hint how to enable vga console in nova 
Issue #3928
 2020-11-23 12:03:00 +01:00
Alexander Boettcher
95c2e5beb3 run: extend pci test to show more information 
Add some more vendors and classcode decoding. Additionally, walk PCI cap
space of each device and decode some of the types.

Issue #3928
 2020-11-23 12:03:00 +01:00
Josef Söntgen
194305a8bb dde_bsd: apply HP sense quirk for Fujitsu S398 
This quirk is needed for properly switching between the internal and
an external headphone.

Thanks to Peter for the patch.

Issue #3929.
 2020-11-23 12:03:00 +01:00
Josef Söntgen
b6912a3d87 dde_bsd: query initial head-phone state 
Account for the situation where the driver is started while the HP
is already plugged in and configure the mixer accordingly.

Thanks to Peter for the patch.

Issue #3929.
 2020-11-23 12:03:00 +01:00
Josef Söntgen
1b4444ce9e dde_bsd: use right member for iterating mixer set 
Due to what seems like a copy-and-paste error, the wrong union member
was used to compare the label. Fortunately, as both structs have the
same memory layout that did not result in any issue.

Issue #3929.
 2020-11-23 12:03:00 +01:00
Josef Söntgen
b9869b666a dde_bsd: use bsd_task to handle I/O 
Originally, the kernel code should have been executed within the
context of the main task like it is done in dde_linux. The initial
port of the HDA driver, however, did not required doing so and the
session called code directly.

When using USB device, that is no longer possible as we have to
suspend the execution during the execution of the kernel code. So
we pass in the audio data and schedule the kernel.

Fixes #3929.
 2020-11-23 12:03:00 +01:00
Josef Söntgen
cd7c99afdc dde_bsd: implemented delayed session announcement 
This becomes necessary in case of the USB audio driver where we have
to wait for and query the function first.

That being said, alternatively we could also announce the session but
ignore any request until there is a device we can use.

Issue #3929.
 2020-11-23 12:02:59 +01:00
Josef Söntgen
2ec398e550 dde_bsd: use timeout framework for time handling 
With this commit the timer back-end uses the timeout framework to
schedule any occuring timeouts and for providing the current time.
For now there is only one timeout, the unsolicited azlia codec event
and therefore the timeout queue consists of solely one timeout object.

In addition a timer session is used for implementing 'delay()' where
we have to block until the delay is completed.

Issue #3929.
 2020-11-23 12:02:59 +01:00
Josef Söntgen
bdb71d94c2 dde_bsd: pull strlcpy in via contrib code 
For historical reason the 'strlcpy' implemention was directly pull in
into the emulation environment. There is, however, no reason not to
use the contrib sources in the usual fashion.

Issue #3929.
 2020-11-23 12:02:59 +01:00
Josef Söntgen
7193902cc0 dde_bsd: properly name PCI audio driver component 
Instead of the generic name, call the PCI driver 'pci_audio_drv'.

This is preliminary clean-up work before introducing the USB audio
driver.

Issue #3929.
 2020-11-23 12:02:59 +01:00
Josef Söntgen
3faf5c43a8 dde_bsd: seperate PCI back end 
The supported drivers so far exclusively used PCI and for practical
reasons the emulation environment was to tied to. To make future
addition of drivers that employ other transport busses easier, split
the bus handling into its own backend.

This is preliminary clean-up work before introducing the USB audio
driver.

Issue #3929.
 2020-11-23 12:02:59 +01:00
Josef Söntgen
6c7f0cb7cc dde_bsd: properly enable recording in run script 
* Updating the run script was missed in commit "dde_bsd: remove
    configration attributes".

  * Decrease output and input volume to "more sane" levels.

Issue #3929.
 2020-11-23 12:02:59 +01:00
Josef Söntgen
54d36a7d1b mixer: apply config on new sessions 
Updating the config and using its settings instead of only reporting
the new session allows to apply stored channel settings immediately.

In case of Sculpt this is useful when an existing launcher already
contains a valid config that sets the volume levels appropiately or
the 'recall_fs' is used.

Fixes #3930.
 2020-11-23 12:02:59 +01:00
Martin Stein
9b164d20fd vfs directory service: name enum Open_mode 
Fixes #3936
 2020-11-23 12:02:59 +01:00
Alexander Boettcher
cd8b436566 usb_hid: avoid destruction of used signal handler 
Fixes #3945
 2020-11-23 12:02:59 +01:00
Piotr Tworek
87e90d640f base-linux: Drop unused member variables 
The _root, _gid and _uid private member variables are not used for
anything. GCC does not care, but clang likes to complain about things
like this.

Issue #3938
 2020-11-23 12:02:59 +01:00
Piotr Tworek
db71cb8c63 base-linux: Drop ununsed Message::_addr 
Not used anywhere in the current codebase. GCC does not care but clang
warns about this.

Issue #3938
 2020-11-23 12:02:59 +01:00
Piotr Tworek
a892018926 base-hw: Define Entry function as noreturn. 
Right now the code marks specific instance of this function as noreturn.
It then tries to initialize it using a value that has the same type,
except for the noreturn part. GCC does not care, but clang complains this
technically assigns the value of entry from incompatible pointer type.

Fix this by defining Entry function as no return.

Issue #3938
 2020-11-23 12:02:59 +01:00
Piotr Tworek
1643d623e4 base-hw: Fix warning about unused lambda capture 
The lambda function passed to Board::Vm_page_table_array does not
actually use "this" for anything. GCC does not warn about this, but
clang does. Remove the extra capture to make clang happy.

Issue #3938
 2020-11-23 12:02:59 +01:00
Piotr Tworek
9b84a8a402 base-hw: Add semicolons after [[fallthrough]] 
GCC doesn't care, but clang complains if [[fallthrough]] is not followed
by a semicolon. Existing code is also not consitent in this regard. Lets
just fix it to make clang happy.

Issue #3938
 2020-11-23 12:02:59 +01:00
Piotr Tworek
db17d51ff1 os: Drop unused _tree member in Root_clock_divider 
Its not used anywhere in the code. Clang produces warning due to this.

Issue #3938
 2020-11-23 12:02:59 +01:00
Piotr Tworek
736b000c19 os: Drop ununsed Main::_id_id variable in ping 
The variable is not used anywhere in the code. This produces warning
when building with clang.

Issue #3938
 2020-11-23 12:02:59 +01:00
Piotr Tworek
187b8ece27 os: Drop ununsed Virtdev_rom::Root::_env member 
Set but not accessed from anywhare. This triggers compilation warning
when building the code with clang.

Issue #3938
 2020-11-23 12:02:59 +01:00
Piotr Tworek
93288bccb3 os: Refer to ::Sandbox::Child explicitly. 
When compiling this code, clang complains reference to 'Sandbox' is
ambiguous. It can either mean ::Sandbox namespace from
os/src/lib/sandbox/types.h, or Genode::Sandbox declared in
repos/os/include/os/sandbox.h. Since the code refers to ::Sandbox::Child
and its already in ::Sandbox namespace we can just drop "Sandbox" and
refer to just Child, which makes clang happy.

Issue #3938
 2020-11-23 12:02:59 +01:00
Piotr Tworek
444bc18fcf base: Drop unused Cpu_root::_md_alloc member 
This private variable has no uses in the context of the class. Spotted
when building the code with clang.

Issue #3938
 2020-11-23 12:02:59 +01:00
Piotr Tworek
18be6315cb base: Drop unused _ram member variable 
Its initialized in constructor but never used. Spotted when building the
code with clang.

Issue #3938
 2020-11-23 12:02:59 +01:00
Piotr Tworek
9c3ce58e57 base: Drop unused "verbose" variable 
Spotted when building the code with clang. This is not used anywhere in
the file.

Issue #3938
 2020-11-23 12:02:59 +01:00
Piotr Tworek
d4a3aa7eda base: explicit copy constructor for Rpc_in_buffer 
According to the "rule of three" [1] and C++11 [2] Rpc_in_buffer needs
to have an explicit copy constructor since it also has user defined
copy assignment operator. Both clang and newer versions of GCC complain
about this.

[1] https://en.cppreference.com/w/cpp/language/rule_of_three
[2] https://www.ece.uvic.ca/~frodo/cppdraft/n4659/html/depr.impldec

Issue #3938
 2020-11-23 12:02:58 +01:00
Piotr Tworek
8d6ca9556f base: fix clang warning for void cast of retval 
This path fixes a void cast used to silence unused return value warning.
Its a common pattern to use void cast to do that. The code uses void *
cast instead. It works for GCC, but clang complains about this.

Issue #3938
 2020-11-23 12:02:58 +01:00
Piotr Tworek
81a49bffee base: exception specification for operator delete 
Clang is rather picky about this and prints the following warning when
compiling new_delete.cc:

  error: function previously declared with an explicit exception
  specification redeclared with an implicit exception specification
  [-Werror,-Wimplicit-exception-spec-mismatch]

Issue #3938
 2020-11-23 12:02:58 +01:00
Piotr Tworek
53a990579b base: Fix UAF in Genode::Pd_session_component::free 
This was discovered when building the code with clang instead of GCC. In
this setup the run/ping on base-hw/arm_v8a/virt_qemu would crash
on shutdown due to uncaught Deref_unconstructed_object exception thrown
for Genode::Reconstructible<Genode::Account<Genode::Ram_quota>>. The
specific instance throwing this exception was
Pd_session_component::_ram_account. My investigation exposed the
following problem:

1. The Pd_session_component has a _sliced_heap member backed by
   _constrained_ram_alloc which in turn uses Pd_session_component itself
   as its Ram_allocator.
2. When ~Pd_session_component is called it first destroys _ram_account,
   followed by _signal_broker.
3. The signal broker holds a reference to
   Pd_session_component::_sliced_heap as Signal_broker::_md_alloc.
4. The base-hw implementation of ~Signal_broker destroys some contexts
   and does this by calling Genode::destroy on some slabs using the
   _md_alloc (ref to Pd_session_component::_sliced_heap).
5. The Genode::Slab calls the Ram_allocator::free which ends up calling
   Pd_session_component::free.
6. The Pd_session_component::free can among other things call replenish
   method on Pd_session_component::_ram_account which has already been
   freed at this point.

From my POV calling replenish at this point is basically an undefined
behavior. The Genode::Constructible holding the Genode::Account was
already detroyed at this point. GCC builds happen to somehow manage to
go through the -> operator call without raising any alarms, while clang
builds trip on the _check_constructed() call.

This fix moves the _ram_account a bit higher in class declaration to
ensure its destroyed after _sliced_heap. This seems like the simpliest
solution for this problem.

Fixes #3941
 2020-11-23 12:02:58 +01:00
Christian Helmuth
a8d3cd9b15 libc: open socket files early on socket creation 
This prevents later file-descriptor shortage when opening files on
demand, which can't be reflected to the application in a sane manner.

The real fix is to open socket files not on libc level but on VFS level
only effectively consume one libc file descriptor for one socket.
 2020-11-23 12:02:58 +01:00
Christian Helmuth
5dfca79bcc libc: use pthread mutex in getifaddrs() 2020-11-23 12:02:58 +01:00
Christian Prochaska
ff429a8056 base-linux: build 'lx_iopl()' function on x86 only 
Fixes #3934
 2020-11-23 12:02:58 +01:00
Christian Prochaska
eafbfb8edf base-linux: socket descriptor caps for RPC (BOARD=pc) 
Fixes #3932
 2020-11-23 12:02:58 +01:00
Christian Prochaska
b72503e581 base-linux: remove 'lx_ioperm()' function 
Fixes #3933
 2020-11-23 12:02:58 +01:00
Stefan Kalkowski
429cd8d37a os: enable smp support for ARM vmm 
* Fix GIC model to support priority and cpu target settings correspondingly
* Fix semantic of SGIR register for GICv2
* Minor GIC model IRQ state fix
* Introduce synchronization for VirtIO and GIC models
* Enable multiple CPUs in test run-script for ARMv8

Fix #3926
 2020-11-23 12:02:58 +01:00
Stefan Kalkowski
6be09a27ca os: minor fix terminal_expect_send test-component 
When an empty line was send, it could occur
that a previous line got printed instead.

Ref #3926
 2020-11-23 12:02:58 +01:00
Stefan Kalkowski
7298b00013 base-hw: make ARMv7/v8 hypervisor smp ready 
* Introduce hypervisor-stack per CPU
* Introduce host world context per CPU
* Mark EL2 translation table memory as inner shareable
* The VMID is not bound to a single VCPU, but to the Vm_session as a whole
* Set affinity of the VCPU accordingly
* Add VMPIDR to VM state

Ref #3926
 2020-11-23 12:02:58 +01:00
Stefan Kalkowski
1d826a2c48 base-hw: do direct syscall when run/pause a VCPU 
Instead of calling core to run/pause a VCPU, go directly to the kernel.
Apart from the performance win, it would otherwise involve a more complex
protocol, when a VCPU on another core has to be removed from the scheduler.
Core's entrypoint handling those request runs on the boot-cpu only.

Ref #3926
 2020-11-23 12:02:58 +01:00
Stefan Kalkowski
40445d7011 base: extend vm_session API with native vcpu cap 
To enable the interaction of a VMM with the kernel directly,
a hidden RPC gets introduced. It allows a kernel-specific
base-library implementation of the Vm_session::Client to request
a kernel-specific capability to address a VCPU, e.g., to
run/stop it.

Ref #3926
 2020-11-23 12:02:58 +01:00
Alexander Boettcher
11e261ada4 platform_drv(x86): use 32bit writes for MSI-X 
Fixes #3927
 2020-11-23 12:02:57 +01:00
Christian Helmuth
c93f3a1136 libusb: establish/terminate USB session 
Now, the USB connection is established on backend initialization and
terminated on backend exit triggered by high-level libusb code.

Thanks to Peter for the patch.
 2020-11-23 12:02:57 +01:00
Christian Helmuth
e339dd542c Prevent warning in test-pthread 2020-11-23 12:02:57 +01:00
Christian Helmuth
3d23c8c419 libports: update curl download location 2020-11-23 12:02:57 +01:00
Martin Stein
89d28c8222 timeout: no volatile stackvars, better warnings 2020-11-23 12:02:57 +01:00
Christian Prochaska
dff3bac441 libc: submit monitor execution signals locally 
Issue #3924
 2020-11-23 12:02:55 +01:00
Christian Prochaska
798beab30e base: support process-local signal submission 
Issue #3923
 2020-11-23 12:02:53 +01:00
Christian Prochaska
50e0f3b977 base: don't throw exceptions in 'Signal_receiver::pending_signal()' 
Issue #3922
 2020-11-23 12:02:49 +01:00
Josef Söntgen
f754e2a7d7 stdcxx: add symbols needed by testsuite 
Fixes #3921
 2020-10-23 15:12:06 +02:00
Josef Söntgen
1dd1bfe692 stdcxx: add new/delete aligned variants (C++17) 
Issue #3921
 2020-10-23 15:08:40 +02:00
Josef Söntgen
a74b572e1f stdcxx: force symlinks to allow for re-preparing 
Issue #3921
 2020-10-23 14:59:56 +02:00
Josef Söntgen
a24911296a stdcxx: adapt header files for testsuite 
Issue #3921
 2020-10-23 14:59:49 +02:00
Josef Söntgen
563cc07cb0 libports: posix src archive implements posix API 
Issue #3921
 2020-10-23 14:58:27 +02:00
Josef Söntgen
59f562f627 libc: add symbols needed by testsuite 
Issue #3921
 2020-10-23 14:58:04 +02:00
Josef Söntgen
4981eb425e ld: add symbols needed by testsuite 
Issue #3921
 2020-10-23 14:58:01 +02:00
Josef Söntgen
de8411a5e1 cxx: add missing low-level symbols 
Issue #3921
 2020-10-23 14:56:36 +02:00
Christian Helmuth
5be1c793a5 depot: update recipe hashes 2020-10-23 14:16:38 +02:00
Emery Hemingway
b4076e762c libc: log a message and exit for raise(...) 
Fix #3919
 2020-10-21 09:14:55 +02:00
Christian Helmuth
6ea628195f dde_bsd: update mirror URL 
It seems RWTH Aachen mirrors only ancient versions currently.
 2020-10-20 09:14:40 +02:00
Martin Stein
64487ded7c timeout: don't warn "timestamp value too big" too often 
Fixes #3657
 2020-10-19 14:26:56 +02:00
Christian Prochaska
405955eaef libc: implement 'posix_memalign()' 
Fixes #3915
 2020-10-15 15:41:55 +02:00
Alexander Boettcher
0aaed47652 nova: re-/store ep fpu state in vm_session 
Fixes #3913
 2020-10-13 14:42:06 +02:00
Alexander Boettcher
20606bc6de foc: save/restore fpu state in vm_session 
Issue #3913
 2020-10-13 14:42:03 +02:00
Christian Prochaska
9cd38a6846 libc: rwlock cleanup 
Issue #3912
 2020-10-13 14:38:43 +02:00
Christian Prochaska
bf4afefaa1 libc: use semaphore in rwlock implementation 
Fixes #3912
 2020-10-13 08:21:04 +02:00
Christian Helmuth
f09b0dc224 Improve synchronization in lwip.run 
The lynx HTTP GET test was started before the HTTP server was up and
therefore failed on Qemu/PBXA9.

Issue #3874
 2020-10-09 16:13:14 +02:00
Christian Prochaska
658030ef49 qt5: update port for qtwebengine improvements 
- unlink shared memory files
- lower maximum number of socket pool sockets to reduce chance of file
  descriptor exhaustion
- fix a build dependency which caused sporadic parallel build errors

Fixes #3910
 2020-10-09 13:51:26 +02:00
Christian Prochaska
4e8bfed5b1 libc: duplicate the file descriptor on shared 'mmap()' mappings 
Fixes #3909
 2020-10-09 13:50:35 +02:00
Christian Prochaska
5c47fa0d41 libc: reduce file descriptor lookup exceptions in 'select()' 
Fixes #3908
 2020-10-09 13:48:23 +02:00
Christian Prochaska
058f2e687c libc: make 'mmap()' address alignment configurable 
With this commit, the alignment of anonymous 'mmap()' allocations can be
configured like this:

<config>
  <libc>
    <mmap align_log2="21"/>
  </libc>
</config>

Fixes #3907
 2020-10-09 13:47:33 +02:00
Josef Söntgen
7d21335ac9 vfs/oss: VFS plugin for Audio_out access via files 
This plugin gives access to the Audio_out session by roughly
implementing a OSS pseudo-device. It merely wrapps the session and does
not provide any resampling or re-coding.

Fixes #3891.
 2020-10-09 13:44:27 +02:00
Josef Söntgen
3d2b0cab93 libc: implement SNDCTL I/O control handling 
In the same vein as the terminal and block I/O controls, the sound
controls are implemented via poperty files and match the OSS
API ([1] features a nice overview while [2] is v3 and [3] gives
in-depth information on the current v4.x API we eventually might want
to implement).

  [1] https://wiki.freebsd.org/RyanBeasley/ioctlref/
  [2] http://www.opensound.com/pguide/oss.pdf
  [3] http://manuals.opensound.com/developer/

The controls currently implemented are the ones used by the cmus OSS
output plugin, which was the driving factor behind the implementation.
It uses the obsolete (v3) API and does not check if the requested
parameter was actually set, which should be done according to the
official OSS documentation.

At the moment it is not possible to set or rather change any
parameters. In case the requested setting differs from the parameters
of the underlying Audio_out session - in contrast to the suggestion in
the OSS manual - we do not silently adjust the parameters returned
to the callee but outright fail the I/O control operation.

The following list contains all currently handled I/O controls.

  * SNDCTL_DSP_CHANNELS sets the number of channels. We return the
    available channels here and return ENOTSUP if it differs from
    the requested number of channels.

  * SNDCTL_DSP_GETOSPACE returns amount of playback data that can
    be written without blocking. For now it amounts the space left
    in the Audio_out packet-stream.

  * SNDCTL_DSP_POST forces playback to start. We do nothing and return
    success.

  * SNDCTL_DSP_RESET is supposed to reset the device when it is
    active before any parameters are changed. We do nothing and return
    success.

  * SNDCTL_DSP_SAMPLESIZE sets the sample size. We return the
    sample size of the underlying Audio_out session and return ENOTSUP
    if it differs from the requested number of channels.

  * SNDCTL_DSP_SETFRAGMENT sets the buffer size hint. We ignore the
    hint and return success.

  * SNDCTL_DSP_SPEED sets the samplerate. For now, we always return
    the rate of the underlying Audio_out session and return ENOTSUP
    if it differs from the requested one.

This commit serves as a starting point for further implementing the
OSS API by exploring more users, e.g. as VirtualBox/Qt5/SDL2 audio
backend or a more sophisticated progam like sndiod.

Issue #3891.
 2020-10-09 13:44:14 +02:00
Christian Helmuth
bcf1cc6397 Ensure high priority for timer in test.run 2020-10-09 13:37:17 +02:00
Martin Stein
bff624c75a test/entrypoint: extend timeout interval 
At least on some PIT-based platforms (x86_32 + pistachio/okl4/sel4), we run
into trouble with the reworked timeout framework that now proccesses all
pending timeouts before calling their handlers. This order change leads to a
higher rate of handling of short periodic timeouts in the timer driver which
can cause lower prioritized components to starve. Especially, if submitting
signals (from timer to client) isn't cheap (as is the case on qemu + pistachio
for example).

Issue #3884
 2020-10-09 13:37:14 +02:00
Martin Stein
512be0a52a test/timer_rate: determine lowest accurate period 2020-10-09 13:37:10 +02:00
Christian Helmuth
91f8281618 depot: update recipe hashes 2020-10-09 13:35:57 +02:00
Christian Prochaska
0e01729d77 libc: handle file descriptor allocation errors 
Fixes #3906
 2020-10-09 13:35:57 +02:00
Sebastian Sumpf
fe1ee05186 recipe: imx53_qsb_drivers 
Do not copy imx53 specfic source code since it has been unified within
the GPIO driver.

follow up for issue #3900
 2020-10-09 13:35:57 +02:00
Christian Helmuth
ec957739e9 Adapt some RAM quotas for 2560x1080 screens 2020-10-09 13:35:57 +02:00
Piotr Tworek
8d5005e03a os: add VirtIO nic driver 
The driver is faily simple and does not support fancy features like
TCP checksum offloading or vlan filtering, but it is fully capable of
running every Genode network based scenario I've tried. Its currently
known to work on virt_qemu arm platforms and x86_64.

Fix #3825
 2020-10-09 13:35:57 +02:00
Piotr Tworek
7fbb245710 os: add VirtIO device helper classes. 
To simplify writing native VirtIO drivers for Genode add helper classes
representing VirtIO device and queue. The queue implementation should
be platform independant. The device abstraction however is closely tied
to the VirtIO transport being used (PCI/MMIO). Both PCI and MMIO
implementations expose the same public API so the actual driver logic
should be the same regardless of which transport is used.

Its also important to note that the PCI version of Virtio::Device
currently does not support MSI-X interrupts. Unfortunately my kowledge
about PCI bus is very limited and my main area of interest was to get
VirtIO drivers working on virt_qemu ARM/Aarch64 platform. As such all
the VirtIO drivers I plan to submit will work with PCI bus, but might
not use some extended capabilities.

Ref #3825
 2020-10-09 13:35:57 +02:00
Piotr Tworek
9bd548c4bd os: add platform_drv config generator for virt_qemu 
The VirtIO device configuration on Qemu is dynamic. The
order and presence of different command line switches affects
base address and interrupt assignment of each device. One could
probably hard-code the necessary switches and resulting XML ARM
platform driver configuration in each run script, but this seems
like troublesome and hard to maintain solution.

This patch explores an alternative approach to the problem.
It implements a ROM driver which probes the address space region
Qemu virt machines assign to VirtIO MMIO devices and exposes the
result as XML via a ROM session. This XML output can be fed directly
as config to the generic ARM platform driver.

Ref #3825
 2020-10-09 13:35:57 +02:00
Piotr Tworek
fe0ad0addb tool: Consolidate qemu nic setup. 
Right now the same code dealing with nic setup on qemu is duplicated
in many different run scripts. It makes it unnecesarily complex to
change the existing config or add support for new nic types. Lets move
all this common code to qemu.inc.

Ref #3825
 2020-10-09 13:35:57 +02:00
Stefan Kalkowski
aa2511e209 platform_drv: make it available for arm_64 generic 
Ref #3825
 2020-10-09 13:35:57 +02:00
Stefan Kalkowski
3cf3344fa3 sel4 + foc: extend port's tool check rules 2020-10-09 13:35:56 +02:00
Sebastian Sumpf
c79687f5f4 gpio: introduce Pin and '_with_gpio' 
- make GPIO server more robust on imx by not throwing exceptions for
  unknown pins, use '_with_gpio' instead
- use 'Gpio::Pin' data type instead of POD 'unsigned'

issue #3900
 2020-10-09 13:35:56 +02:00
Sebastian Sumpf
b9bd179e54 gpio_drv: remove specs for imx53 and imx6 
Also cleanup run scripts and recipes were gpio driver is not required,
update the ones were it is.

issue #3900
 2020-10-09 13:35:56 +02:00
Sebastian Sumpf
6c6deb7e8b dde_linux: add touch to i.MX8 recipe 
Add touch screen and event filter to drivers interactive package.

issue #3900
 2020-10-09 13:35:56 +02:00
Sebastian Sumpf
d387eba0ba os: driver recipe for i.MX8 drivers 
Includes touch and gpio driver

issue #3900
 2020-10-09 13:35:56 +02:00
Sebastian Sumpf
96eb83f19a os: Synaptics DSX touch screen driver 
Provides touch event service for i.MX8M Quad only

issue #3900
 2020-10-09 13:35:56 +02:00
Sebastian Sumpf
89972b11b7 gpio_drv: use platform session for i.MX 
This makes the driver work for all i.MX boards. A platform driver with
the board specific configuration is required.

issue #3900
 2020-10-09 13:35:56 +02:00
Sebastian Sumpf
664b861f9d imx8_fb_drv: MIPI DSI display support 
- support for Northwest Logic MIPI DSI bridge
- support for Radium panel (NXP MX8_DSI_OLED1)
- new 'DSI-1' connector in driver config file

issue #3900
 2020-10-09 13:35:56 +02:00
Sebastian Sumpf
27f705bc48 bootstrap: iomux adaptions for i.MX8 MIPI 
These settings were obtained with a JTAG debugger from a running Linux
system. They work for MIPI as well as for HDMI.

issue #3900
 2020-10-09 13:35:56 +02:00
Alexander Boettcher
325e9cb9fa platform_drv(x86): avoid memory leak 2020-10-09 13:35:56 +02:00
Alexander Boettcher
50b10ef4a5 aes_cbc_4k: extend test and run in autopilot 
- to compare crypted text with host openssl

Fixes #3905
 2020-10-09 13:35:56 +02:00
Alexander Boettcher
c0f8022a78 aes_cbc_4k: use libcrypto 
Issue #3905
 2020-10-09 13:35:56 +02:00
Alexander Boettcher
5d808cdc01 platform_drv: extend MSI-X supports 
The patch handles the case, that the memory for the MSI-X table is part
of one of the Pci::Resource Memory BARs, which got allocated beforehand already.

With this commit, the platform driver will not fall back to use legacy IRQs or MSI, whereby MSI-X is available actually. Additionally, this patch avoids a lot of red
messages about non available IO-MEM printed by the roottask.

Fixes #3904
 2020-10-09 13:35:56 +02:00
Christian Helmuth
abefca500b libc: fix deadlock in pthread_cond_timedwait/signal() 
The deadlock occured with three concurrently running threads: two
waiters calling pthread_cond_timedwait() and one signaller calling
pthread_cond_signal().

If waiter W1 hits its timeout, the signaller may have called
pthread_cond_signal(), detected this waiter and posted the internal
'signal_sem' concurrently. Then, the signaller waits for 'handshake_sem'
to ensure the waiter got woken up.

Waiter W1 can't consume the 'signal_sem' post by
'sem_wait(&c->signal_sem)' because another waiter W2 may have consumed
the post already above in sem_wait/timedwait(). Waiting for a post on
'signal_sem' would block the waiter W1 in perfect deadlock with
signaller on 'handshake_sem'. As W1 also owns 'counter_mutex' in this
situation, waiter W2 would block when trying to aquire 'counter_mutex'
and can't resolve the situation.

So, W1 does nothing in this case and we accept the spurious wakeup on
next pthread_cond_wait/timedwait().
 2020-10-09 13:35:56 +02:00
Martin Stein
7feea78991 timeout: rework timeout framework 
* get rid of alarm abstraction
* get rid of Timeout::Time type
* get rid of pointer arguments
* get rid of _discard_timeout indirection
* get rid of 65th bit in stored time values
* get rid of Timeout_scheduler interface
* get rid of uninitialized deadlines
* get rid of default arguments
* get rid of Timeout::_periodic
* get rid of Timeout::Raw
* use list abstraction
* only one interface for timeout handlers
* rework locking scheme to be smp safe
* move all method definitions to CC file
* name mutexes more accurate
* fix when & how to set time-source timeout
* fix deadlocks

Fixes #3884
 2020-10-09 13:35:56 +02:00
Martin Stein
9e5d479d03 timeout: test smp support 
Ref #3884
 2020-10-09 13:35:56 +02:00
Christian Helmuth
26011a7151 libc: update status for component select handlers 
In case of contexts blocked in select() the monitor updates the
file-descriptor status, but if the entrypoint is just blocked for the
select handler, the status must be updated explicitly on
dispatch_select().
 2020-10-09 13:35:56 +02:00
Stefan Kalkowski
bbb017dc24 muen: update to Community 2019 toolchain version 
Fix #3903
 2020-10-09 13:35:55 +02:00
Alexander Boettcher
04d3c9e750 nova: re-enable DMAR super pages in kernel 
Fixes #3902
 2020-10-09 13:35:55 +02:00
Alexander Boettcher
e5fe9c6fc7 qemu-usb: catch exception on already gone devices 
Fixes #3893
 2020-10-09 13:35:55 +02:00
Norman Feske
04821b1abc nitpicker: apply focus change w/o input events 
This patch fixes the corner case where the keyboard focus is defined
independently from user interactivity, e.g., the activation of a
screensaver or lock screen.

In this case, nitpicker would update its internal focus state not before
the next input event is handled. Should this input event be a press
event, this event would wrongly be delivered to the prior focused
session. Another problematic situation is the initial state before the
very first input event occurs. Since the focus remains undefined until
the first input event is handled, an initial key press event would not
be delivered.

This is a regression caused be the transition to the event-session
interface and the removal of the nitpicker's periodic way of operation.
The patch fixes the problem by applying pending focus changes not only
at the input processing but also on the code path that responds to focus
changes (e.g., focus-rom update).

Issue #3812
 2020-10-09 13:35:55 +02:00
Alexander Boettcher
afab15f1a4 linux: round up dataspace to 4k sizes 
Issue #3901
 2020-10-09 13:35:55 +02:00
Alexander Boettcher
e61f6cfd38 base: add thread migration test 
Issue #3842
 2020-10-09 13:35:55 +02:00
Alexander Boettcher
90bea1499e core: store new affinity on successful migration 
Adjust the base-* platforms to acknowledge new thread location solely if
migration is supported and succeeded. Otherwise the wrong thread
locations are observed via the trace session and utilization time calculation
get wrong.

Issue #3842
 2020-10-09 13:33:36 +02:00
Alexander Boettcher
99fa203673 nova: add migration support for global threads 
Fixes #3842
 2020-10-09 13:33:36 +02:00
Martin Stein
1b41d9db90 base: remove alarm library from base 
Ref #3884
 2020-10-09 13:33:36 +02:00
Martin Stein
c1d0179194 dde_rump: move private headers to src/include 
Ref #3884
 2020-10-09 13:33:36 +02:00
Martin Stein
04463806a8 dde_rump: remove unused rump_cgd 
Ref #3884
 2020-10-09 13:33:36 +02:00
Martin Stein
af01370cc1 run/rump_fat: raise ram quota 
Ref #3884
 2020-10-09 13:33:35 +02:00
Martin Stein
4eb4bd6f96 trace_logger: warn on multiple buffer wraps 
Ref #3884
 2020-10-09 13:33:35 +02:00
Christian Prochaska
d2d74cc5fa tool chain: fix build error on Ubuntu 20.04 (GCC 9) 
Fixes #3899
 2020-10-09 13:33:35 +02:00
Josef Söntgen
f53df495db libc: provide 'sys/soundcard.h' header file 
Issue #3891.
 2020-10-09 13:33:35 +02:00
Josef Söntgen
f3268cade6 libc: split ioctl method 
Use one ioctl method for each type of I/O control because by now the
general method will become increasingly long.

Fixes #3890.
 2020-10-09 13:33:35 +02:00
Josef Söntgen
1a54ee895e vfs: use compound ioctl in Block-VFS plugin 
Instead of using the old 'ioctl' Vfs::File_io_services API implement
the I/O control functionality in a buch of files. This is similar to
the terminal-VFS plugin.

Fixes #3889.
 2020-10-09 13:33:35 +02:00
Josef Söntgen
27d4cb871f libc: use property files for block ioctl 
Like already done for terminal I/O controls use collect the information
by reading property files instead of using the old VFS ioctl interface.

Fixes #3888.
 2020-10-09 13:33:35 +02:00
Josef Söntgen
2312ad35dd libc: match ioctl request type to contrib type 
There is a type mismatch as in the FreeBSD contrib code the type of the
request is 'unsigned long'. So far, only I/O controls where the request
falls into the signed range where used and this was not a problem.

Some of the SNDCTL requests, however, have the bit set.

Fixes #3887.
 2020-10-09 13:33:35 +02:00
Christian Prochaska
85a84f5042 qt5: generate enter events related to popup menus 
Fixes #3894
 2020-09-18 15:23:45 +02:00
Sebastian Sumpf
0fd979b147 imx8_fb_drv: make 'Capture' constructible 
Reconstruct capture connection on config update in order to receive
update of complete screen data from GUI session.

fixes #3878
 2020-09-18 15:21:10 +02:00
Christian Helmuth
ad595d2701 sculpt: version 20.09 2020-09-18 15:20:18 +02:00
Christian Helmuth
f6337a6446 depot: update recipe hashes 2020-09-18 14:04:56 +02:00
Stefan Kalkowski
f1b3e826d5 sculpt: update building section of sculpt README 2020-09-18 14:04:56 +02:00
Alexander Boettcher
2afba3c137 driver_manager: provide more memory to boot_fb 
Fixes #3896
 2020-09-18 14:04:56 +02:00
Norman Feske
e0d9a04f67 libc: resolve circular dependency in fork 
This patch is a follow-up for the commit "libc: use monitor for fork".
It removes the use of the monitor mechanism from the
'Local_clone_service::close' RPC function because the fork_ep must stay
responsive for the destruction and creation of 'Child' objects.

Issue #3874
 2020-09-18 14:04:56 +02:00
Norman Feske
274f306315 release_notes-20-08.txt: minor style fixes 2020-09-18 14:04:56 +02:00
Christian Helmuth
c59c266afc depot: update recipe hashes 2020-09-17 14:23:14 +02:00
Norman Feske
9aca1ac775 News item for Sculpt 20.08 2020-09-17 14:01:19 +02:00
Norman Feske
3e83b4b39e Update Sculpt documentation for version 20.08 2020-09-17 14:01:19 +02:00
Stefan Kalkowski
d7eb174c88 muen: fix building on modern devel distros 
* Use python2 interpreter not available by default explicitely
* Update ada-bfd binding library to meet binutils-dev >= 2.34
 2020-09-17 10:14:07 +02:00
Stefan Kalkowski
c48b3ca16f okl4: use python2 explicitely in elfweaver tool 
Fix #3892
 2020-09-17 10:14:07 +02:00
Norman Feske
951409f14b intel_fb: reinit capture session on mode change 
The current version of the driver leaves the screen in a black state
after a mode change, e.g., when connecting an external display.
With this patch, the framebuffer content is fetched for the entire
screen after a mode change. Thanks to Sebastian Sumpf for investigating.

Issue #3878
 2020-09-17 10:14:07 +02:00
Norman Feske
2075b119ac sculpt: usability tweaks in popup dialog 
- Show "..." for resource-assignment menu entry because it leads
  to a sub menu.

- Avoid dialog changes when clicking on "Add component" while a
  routing or resource dialog is unfolded.
 2020-09-17 10:14:07 +02:00
Norman Feske
0d61029d7e depot_deploy: move affinity to <affinity> sub node 
The initial implementation of the affinity configuration in
"depot_deploy: support affinity configuration" added the affinity
location attributes to the <start> node of the deploy config. This patch
moves the information into a dedicated <affinity> sub node as done by
the init configuration. So the context of the attributes 'xpos', 'ypos',
'width' and 'height' becomes clear.

It also fixes a usability issue in Sculpt that occurred during testing:
When configuring multiple components with custom affinities, the
resources dialog of later components would wrongly display the state of
earlier components instead of displaying the fresh (default) state. The
resulting configuration would then not match the displayed information.
This is fixed by resetting the dialog state.

As another minor cosmetic change, the patch adds a line break in front
of copied <config> or <heartbeat> nodes.

Issue #3597
 2020-09-17 10:14:07 +02:00
Christian Helmuth
2eb8c5e21a net: move ascii_to() into Net namespace 
The combination of Net::Mac_address and
Genode::ascii_to(Net::Mac_address) required shaky quirks in several
places because GCC is not able to resolve the ascii_to overload if
base/xml_node.h was included to early. The current solution moves the
several ascii_to overloads "closer" to the Net types by putting them
into the Net namespace, where GCC reliably picks them up.

Hence, co-locating the ascii_to() utility with the overload type in the
same scope/namespace is good practice.

This patch removes the now obsolete <nic/xml_node.h> header file.
 2020-09-17 10:13:22 +02:00
Norman Feske
c0f9f2c6d7 sculpt: restore toggleable format button 
With the move of the storage-management dialogs to the graph in Sculpt
version 20.02, the ability to unselect the buttons for destructive
storage operations such as format or expand disappeared. This patch
restores the original behavior.
 2020-09-17 10:13:21 +02:00
Alexander Boettcher
f0f8d0e0ca base: add affinity to session creation request 
Fixes #3838
 2020-09-17 10:13:21 +02:00
Norman Feske
5dc8e330b6 sculpt: add button for restarting a component 
Fixes #3885
 2020-09-17 10:13:21 +02:00
Alexander Boettcher
c9f2847420 foc: make serial esc functional 2020-09-17 10:13:21 +02:00
Stefan Kalkowski
fcb0ca305c dde_linux: don't zero out framebuffer of imx8 
* On this platform there is no need to zero out the framebuffer
  dataspace, which is already blanked by core
* But it might happen that the GUI server is sending a capture
  event before the zero out happens. Thereby screen content can get
  lost

Fix #3878
 2020-09-17 10:13:21 +02:00
Stefan Kalkowski
fd161cd814 dde_linux: turn imx8_fb_drv into Platform client 
* Requests its I/O resources and clock settings from the platform driver

Fix #3877
 2020-09-17 10:13:21 +02:00
Stefan Kalkowski
5913cdae89 os: fix clock settings for i.MX 8M platform_drv 
* Fixes faulty algorithm in fractional pll rate calculation
* Enables clocks that are set as reference clock to prevent system freeze
* Enables/disables root clocks of gates implicitely

Fix #3876
 2020-09-17 10:13:21 +02:00
Norman Feske
8dde14f93e bash: prevent aliasing of libc's 'getenv' 
By default, bash brings its own version of 'getenv', named exactly like
the libc function. This becomes a problem in fork/execve scenarios if
the dynamic linker resolves 'getenv' to the bash binary instead of the
libc.

This patch fixes the generate step of the genodians.org scenario.

Issue #3882
 2020-09-17 10:13:20 +02:00
Norman Feske
f5dc71ed35 os: remove obsolete single_session_service.h 2020-09-17 10:13:20 +02:00
Christian Prochaska
e0ca250232 libc: support detached pthreads 
Fixes #3880
 2020-09-17 10:13:20 +02:00
Christian Helmuth
75ba52a52b libc: remove mutex parameter from monitor 
Issue #3874
 2020-09-17 10:13:20 +02:00
Christian Helmuth
a0a112ffe7 libc: use monitor for fork 
Issue #3874
 2020-09-17 10:13:20 +02:00
Christian Helmuth
a891b3832c libc: use monitor for pthread join/cancel 
Issue #3874
 2020-09-17 10:13:20 +02:00
Christian Helmuth
d6f89b285d libc: use monitor for sleep() 
Issue #3874
 2020-09-17 10:13:20 +02:00
Norman Feske
e9a3f0f095 libc: use monitor for select 
This patch simplifies the control flow within the libc kernel by
facilitating the monitor mechanism for the implementation of 'select'.

Issue #3874
 2020-09-17 10:13:20 +02:00
Christian Helmuth
52a6cf1412 libc: support creat() 2020-09-09 16:57:34 +02:00
Norman Feske
aeeade53f9 sculpt: don't restart menu view if font changes 
This patch enables the menu view's new ability to respond to font-size
changes in Sculpt so that the menu view instanced no longer need to be
restarted whenever the screen resolution or font size is changed.

Fixes #3875
 2020-09-09 16:57:34 +02:00
Norman Feske
c05f716478 menu view: respond to font configuration changes 
Issue #3875
 2020-09-09 16:57:34 +02:00
Norman Feske
4aca94b08b vfs/ttf: support for watching font-size changes 
This patch allows the use of the VFS watch mechanism for the glyph file
of the TTF VFS plugin so that clients become able to dynamically respond
to font reconfigurations.

Issue #3875
 2020-09-09 16:57:34 +02:00
Norman Feske
3031fd2a7d os/vfs.h: const 'Directory' arg for 'Watch_handler' 
The watch mechanism should not require a reference to a mutable
directory.
 2020-09-09 16:57:34 +02:00
Norman Feske
7de2b040f8 vfs: deliver ACKs after config change 
This is needed to enable VFS plugins to notify VFS clients about file
changes that depend on the plugin configuration, E.g., whenever the vfs_ttf
plugin responds to a font-size change, it generates a watch notification
for the glyphs file. Since the change is independent from I/O, we need
to manually call 'handle_io_progress'.
 2020-09-09 16:57:34 +02:00
Norman Feske
0af969543d nitpicker: preserve alpha channel during resize 
This patch reduces flickering artifacts that appear during the
resizing of clients that use an alpha channel.

Issue #3812
 2020-09-09 16:57:34 +02:00
Josef Söntgen
9e97393e3a os: check path for dataspace in ROM VFS plugin 
Fixes #3879.
 2020-09-09 16:57:34 +02:00
Norman Feske
be14e68a83 sculpt/vimrc: 'set nowritebackup' 
By default, Vim renames a written file to a backup file suffixed with
"~" before writing the current buffer to a new file. Consequently, there
exists an intermediate state when no file exists. Should a client watch
such a file to obtain dynamic configuration info, it observes the empty
state.

Some components have builtin heuristics for such a situation. In
particular, the window layouter falls back to a predefined default
'rules' if no rules are provided as a file. So when interactively
editing window-layouter rules using Vim, it can happen that the manually
maintained rules get overwritten by the default rules.

By setting 'set nowritebackup', we can sidestep this issue by preventing
Vim from producing the bad intermediate state.
 2020-09-09 16:57:34 +02:00
Stefan Kalkowski
c1d99630c2 base: reset discarded Alarm object appropriatedly 
Fix #3881
 2020-09-09 16:57:34 +02:00
Norman Feske
328a4fa644 nitpicker: update hovering on view-stack changes 
With the change of nitpicker to the event session interface, the
formerly periodic hovering updates moved to the - now sporadic - input
processing. This has the unfortunate side effect that hovering changes
caused by non-user-input, in particular view-stack changes issued by the
GUI clients that change the view under the current pointer position,
would no longer be reported immediately but only after receiving the
next incoming input event.

This patch reworks the hover handling such that potential hovering
changes due to view-stack operations are evaluated immediately by
those operations, covering the update of the hover report and the
generation of artificial enter/motion events.

Issue #3812
 2020-09-09 16:57:34 +02:00
Christian Helmuth
ff82dc1ad5 doc: activate --depot-auto-update in getting_started.txt 
Also be more explicit about the command to "execute" if depot archives
are missing in depot.inc.
 2020-09-09 16:57:34 +02:00
Norman Feske
17a6318ad6 vfs: process all acks in each iteration 
This patch solves a corner case where one long-active job (e.g.,
read-ready request) stays at the beginning of the '_active_jobs' queue
without an ack. In this case, the '_try_acknowledge_jobs' method would
wrongly stop processing the subsequent acknowledgements. In practice,
this can lead to a delayed sending of acknowledgements until new I/O or
client requests occur. In particular, Vim in Sculpt's inspect window
sometimes did not immediately respond to key presses during tab
completion. Here, the read-ready request of the terminal prevented the
acknowledgement for read of directory entry from being delivered until
the next key was pressed.

Fixes #3873
 2020-09-09 16:57:34 +02:00
Christian Helmuth
0605180a61 libc: remove legacy kernel functions 2020-09-09 16:57:34 +02:00
Christian Helmuth
ab953a534c libc: remove residues of noux fork support 2020-09-09 16:57:34 +02:00
Christian Helmuth
0977574372 vfs/rtc: return timestamp size on stat() 2020-09-01 16:48:50 +02:00
Alexander Boettcher
99bdfbe36f trace: don't account argument_buffer 2x 
The quota for the argument buffer is already accounted by using the
Attached_ram_dataspace _argument_buffer, which uses the Constraint_ram_allocator
_ram, which uses the Ram_quota_guard from the Session_object. Running on
Sculpt with more than 1000 Subject_info objects/trace IDs the memory
waste become noticeable.
 2020-09-01 16:48:29 +02:00
Christian Helmuth
f70b02ae3b vbox: remove logger libc plugin 
Fixes #3870
 2020-08-28 14:48:05 +02:00
Christian Helmuth
574e41119a sculpt: version 20.08 2020-08-28 14:34:22 +02:00
Christian Helmuth
8d31590a66 version: 20.08 2020-08-28 10:31:12 +02:00
Norman Feske
8a5277d53f News item for version 20.08 2020-08-28 10:18:07 +02:00
Norman Feske
fb6549a092 Release notes for version 20.08 2020-08-28 10:18:07 +02:00
Christian Helmuth
c649307720 depot: update recipe hashes 2020-08-28 08:29:12 +02:00
Christian Helmuth
bf0f3b65b4 Replace remaining 'constrain_phys' by 'managing_system' 2020-08-28 08:28:13 +02:00
Norman Feske
b9a21ee3ae nitpicker: updated README 2020-08-28 08:28:13 +02:00
Piotr Tworek
8d790010bf Xml_generator: Remove unused member variable 
This triggers a warning when building the code with clang.

Fixes #3868
 2020-08-28 08:28:13 +02:00
Stefan Kalkowski
e29485fa59 dde_linux: decouple Lx_kit::Irq and Platform::Device 
Follow-up commit to fix the old usb_drv. Under rpi one and the
same irq can be requested several times. Therefore, we've to track
the Irq_connection objects.

Ref #3865
 2020-08-28 08:28:13 +02:00
Christian Prochaska
ace172ebf3 qt5: prerequisites for qtwebengine support 
Fixes #3867
 2020-08-28 08:28:13 +02:00
Norman Feske
bbfda019df vfs/terminal: minor const improvement 
Issue #3860
 2020-08-25 12:03:31 +02:00
Norman Feske
5cc2adb421 pkg/sculpt: update README 
Adaptation to the change of the input filter to the event filter.
 2020-08-25 12:02:37 +02:00
Norman Feske
90b3b72a91 doc: update components.txt 
This patch adjusts the components.txt to the recent changes of the GUI
stack.

Issue #3812
 2020-08-25 12:02:36 +02:00
Christian Helmuth
dd4a3b0263 vbox: direct screen updates via NotifyUpdate() 2020-08-25 12:02:36 +02:00
Norman Feske
7d0cb9620b depot: update recipe hashes 2020-08-25 11:50:41 +02:00
Christian Helmuth
2c29bf5a21 tool_chain_auto: depend on zlib from depot 
This prevents run-script failures in case hashes are out-of-date.
 2020-08-25 11:50:12 +02:00
Piotr Tworek
80bc1cff5f run: Use Qemu 5.0.0 compatile gic-version param. 
It seems Qemu 5.0.0 no longer supports gic_version parameter.
Apparently this only worked due to some compat feature that
was dropped in Qemu 5.0.0. The replacement which works on both
old and new Qemu versions is gic-version.

Fixes #3823
 2020-08-25 11:50:12 +02:00
Stefan Kalkowski
a5f6d0f081 dde_linux: use new platform API for usb_host_drv 
Introduce a common platform device initialization routine for all
ARM SoCs by using the new platform driver API.

Fix #3865
 2020-08-25 11:50:12 +02:00
Stefan Kalkowski
e223be32ce dde_linux: decouple Lx_kit::Irq and Platform::Device 
The Lx_kit::Irq abstraction of DDE Linux was always using the very first
interrupt of a Platform::Device stored in it. Instead of handing over the
platform device it is much more flexible and sane to hand over the actual
interrupt capability.

Ref #3865
 2020-08-25 11:50:12 +02:00
Stefan Kalkowski
73d3698e2f os: add new platform driver for Raspberry Pi 
Fix #3864
 2020-08-25 11:50:12 +02:00
Stefan Kalkowski
5f5ad41ad3 hw: unify irq enumeration for Raspberri Pi 
By now, the enumeration of peripheral interrupts on Raspberry Pi 1 was
different in between base-hw kernel and Fiasco.OC. Therefore, hacks were
needed in every driver to request the correct interrupt number dependent
on the kernel. Before reproducing the same in the platform driver for rpi,
we can more easily use the same enumeration with base-hw.

Ref #3864
 2020-08-25 11:50:12 +02:00
Stefan Kalkowski
7ba31d4447 os: disable initialization of i.MX8MQ platform_drv 
This is a temporary workaround to not harm drivers, which aren't
converted yet to request all device resources including power and
clocks from this new platform driver.

Ref #3863
 2020-08-25 11:50:12 +02:00
Stefan Kalkowski
224f5907b2 os: add platform driver for i.MX 8M Quad SoC 
The new platform driver supports clock and power-domain settings per device.
IOMUX constroller settings are not part of the driver yet.

Fix #3863
 2020-08-25 11:50:12 +02:00
Stefan Kalkowski
70f98fcc44 hw: implement Pd_session::managing_system for ATF 
To access the ARM Trusted Firmware from the platform driver
fill the new `managing_system` call of the PD session with life resp.
do a SMC call on behalf of the client.

Fix #3816
 2020-08-25 11:50:11 +02:00
Stefan Kalkowski
06edc0d52b base: extend PD session with managing_system call 
Introduce the managing_system privilege for components like the
platform_driver to allow it to call system management functionality
that is reserved by kernel or special firmware, e.g., ARM Trusted Firmware.

The former RAM resource configuration attribute `constrain_phys`,
which enabled to constrain the region of physical RAM to be used,
gets replaced by the new, broader managing_system configuration
attribute of a `start` node. It gets enforced by the sandbox library.

Ref #3816
 2020-08-25 11:50:11 +02:00
Christian Prochaska
af6d2a8c54 stdcxx: add abi symbols needed by qtwebengine 
Fixes #3862
 2020-08-25 11:50:11 +02:00
Josef Söntgen
157f4b1270 pthread: implement condattr setclock 
Normally CLOCK_REALTIME is used. However libraries, like glib, want to
use CLOCK_MONOTONIC. To make those users happy add setting the clock.

Note, the pthread_cond implementation uses the POSIX semaphore API
internally that does not have means to set the clock. For this reason,
the private 'sem_set_clock' function is introduced.

Fixes #3846.
 2020-08-25 11:50:11 +02:00
Martin Stein
66ff18a53e libc: support pthread mutex type "Adaptive NP" 
The initial motivation for enabling support for this pthread mutex type was
that it is required by the Glib test in genode/world.

Fixes #3817
 2020-08-25 11:50:11 +02:00
Norman Feske
2a4adc895c Remove dithering from menu_view, themed decorator 
With the change from RGB565 to RGB888, the application of dithering has
become obsolete.

Issue #3784
 2020-08-25 11:50:11 +02:00
Josef Söntgen
d21bc8268b vfs: add raw option to terminal plugin 
When the 'raw' attribute is set to 'yes' the terminal VFS plugin will
ignore control characters. This in necessary for terminal connections
that transport data that contain such characters as part of message.

Fixes #3860.
 2020-08-25 11:50:11 +02:00
Norman Feske
8b556a9435 usb_block: fix upper block-number range check 
The driver wrongly rejected a block request for the very last block of
the device, which prevented part_block from successfully parsing the
partition table (when attempting to access the GPT backup).

Fixes #3861
 2020-08-25 11:50:11 +02:00
Norman Feske
a68a6665ac sculpt: remove part_block upon failure 
With this patch, sculpt uses init's heartbeat-monitoring mechanism to
detect the failure of part_block instances during storage disovery.
If part_block gets stuck, the device is released and can thereby
be accessed at the whole-device level.

Issue #3861
 2020-08-25 11:50:11 +02:00
Norman Feske
1e7c94759d part_block: spelling fix 2020-08-25 11:50:11 +02:00
Christian Prochaska
9ec2a19cc0 libc: add wrapper function for 'Genode::cache_coherent()' 
Fixes #3858
 2020-08-25 11:50:11 +02:00
Christian Prochaska
9bc6b8be5a libc: handle 'MAP_SHARED' flag in 'mmap()' 
Fixes #3857
 2020-08-25 11:50:11 +02:00
Norman Feske
f8f8b665c3 vfs: don't accept new job before old job is acked 
Fixes #3859
 2020-08-25 11:50:11 +02:00
Christian Prochaska
31397d67ae base: increase linker area size 
Fixes #3856
 2020-08-25 11:50:11 +02:00
Christian Prochaska
dd8777093d libc: don't treat 'mmap()' address hint w/o MAP_FIXED flag as error 
Fixes #3855
 2020-08-25 11:50:10 +02:00
Christian Prochaska
2f55ffdf20 libc: zero-initialize anonymous memory from 'mmap()' 
Fixes #3854
 2020-08-25 11:50:10 +02:00
Josef Söntgen
7c1a4522d6 virtualbox5: enable storage device flushing 
The patches disable VBOX_IGNORE_FLUSH and a sanity check in the VMDK
backend. This enables passing an explicit flush request by the guest
down to the VFS.

Fixes #3743.
 2020-08-25 11:50:10 +02:00
Norman Feske
3e8824908d system_shell: increase gui_fb quota 
With this change, the window can be maximized when using a 1920x1080
resolution with 32 bits per pixel.

Issue #3784
 2020-08-25 11:50:10 +02:00
Norman Feske
28714979bd sculpt: drop storage-detect-cancellation feature 
Fixes #3853
 2020-08-25 11:50:10 +02:00
Norman Feske
658091bfad ttf_font: add sanity check for invalid scale value 
The check prevents the Ttf_font from violating the bounding box in the
presence of very small scale values. This can happen during the startup
of Sculpt. Before the framebuffer driver is up, Sculpt bases its dynamic
font-size setting on a screen resolution of 1x1.

Issue #3812
 2020-08-25 11:50:10 +02:00
Norman Feske
e8e14ad1bf test-libc_execve: adjust timeout for pistachio 2020-08-25 11:50:10 +02:00
Martin Stein
e544464354 libsparkcrypto: src recipe, contrib alis, mk fixes 
* add libsparkcrypto source-recipe
* provide ALI files through a new repository by mstein and add the repo
  download to the libsparcrypto port-file
* remove dependencies to non-existent contrib ADB files from the library make
  files

Fixes: #3852
 2020-08-25 11:50:10 +02:00
Norman Feske
969e59c599 terminal: consider border in size calculation 2020-08-25 11:50:10 +02:00
Norman Feske
f2d7f7aa6e sculpt: remove decor frame from panel view 2020-08-25 11:50:10 +02:00
Norman Feske
a59b69758b sculpt: remove deferred GUI initialization 
The sculpt manager used to defer the initialization of the GUI until
nitpicker's first display report became ready. This way, Sculpt was able
to run headlessly even if the framebuffer driver failed to start up.
Thanks to #3827, nitpicker no longer depends on a working framebuffer
driver. So the sculpt manager is safe to rely on nitpicker in any case,
simplifying the code.

Issue #3827
 2020-08-25 11:50:10 +02:00
Norman Feske
c7d9b2ca92 nitpicker: fix mode change for requested fb 
This is a follow-up commit to "nitpicker: make framebuffer and input
optional". It restores the dynamic mode-change support when using
'request_framebuffer="yes"' as needed in scenarios where multiple
nitpicker instances are used in a cascaded way. E.g., Sculpt's
Leitzentrale. The previous version missed to reconstruct the
'_fb_screen' on mode changes.

Issue #3812
 2020-08-25 11:50:10 +02:00
Norman Feske
2437d759b6 Xml_node: remove deprecated methods 
Issue #3755
 2020-08-25 11:50:10 +02:00
Norman Feske
0434cb6fd6 gdb_monitor.run: cap-quota adjustment 2020-08-25 11:50:10 +02:00
Norman Feske
556a5c8086 libc: clean up 'clock_gettime' handling 
This patch untangles the dependency of VFS operations that need RTC
information from the 'clock_gettime' libc function that must never be
called from the libc kernel context.

- The 'Rtc' class uses the VFS directly for reading the rtc file instead
  of relying on libc functions.

- The 'Rtc' instance has become part of the 'Kernel' instead of
  being construced as a side effect of the first call of
  'clock_gettime'.

- Changed 'Rtc::read' to return a timespec value, which has a higher
  precision than the formerly used time_t value.

- The 'Rtc::read' returns a value with the relative 'current_time'
  already applied. The former handling of subsequent rtc-value
  updates has been rewritten to become more logical.

- The 'Vfs_plugin' no longer calls 'clock_gettime' but the new
  kernel-level 'Current_real_time' interface.

Issue #2635
 2020-08-25 11:50:10 +02:00
Norman Feske
852ab79359 Move atexit handling from base lib to libc 
This patch untangles the interplay of the base library and the libc
during the exit handling.

- The CXA ABI for the atexit handling is now provided by the libc.
  For plain Genode components without libc dependency, __cxa_atexit
  is a no-op, which is consistent with Genode's notion of components.

- The 'abort' implementation of the base library no longer calls
  'genode_exit' but merely 'sleep_forever'. This way, the cxx library
  no longer depends on a 'genode_exit' implementation.

- The libc provides 'atexit' support by storing metadata on the
  libc kernel's heap now, thereby eliminating the former bounded
  maximum number of atexit handlers.

- Shared-library dtors are no longer called via the atexit mechanism
  by explicitly by the dynamic linker. This slightly changes the
  call order of destructors (adjustment of the ldso test). Functions
  marked as destructors are called after the atexit handlers now.

- The libc executes atexit handlers in the application context,
  which supports the I/O operations in those handles, in particular
  the closing of file descriptors.

Fixes #3851
 2020-08-25 11:50:09 +02:00
Christian Prochaska
66063e5137 qt5_component: support arguments and environment from config 
Fixes #3049
 2020-08-25 11:50:09 +02:00
Josef Söntgen
f03917ab7c aes_cbc_4k: api recipe 
Fixes #3850
 2020-08-25 11:50:09 +02:00
Martin Stein
5ed528a2ad aes_cbc_4k: import file for include directories 
Ref #3850
 2020-08-25 11:50:09 +02:00
Josef Söntgen
81ae4599ae libsparkcypto: api recipe 
For the time being, we put all spec and body files into the recipe although
there might be a (so far to us unknown) way to have only the API relevant files
in it.

Fixes #3849
 2020-08-25 11:50:09 +02:00
Martin Stein
82090d2ea1 libsparkcrypto: mark some units pure 
* switch to a libsparkcrypto fork of m-stein
* switch to a libsparkcrypto state that makes some units pure that are used by
  the consistent block encrypter, in order that the latter can become
  completely pure

Ref #3849
 2020-08-25 11:50:09 +02:00
Martin Stein
2e3b11b354 ada-runtime: add s-expint.ad* 
* switch to fork of the Ada runtime provided by m-stein
* switch to a Ada-runtime state that provides the Exp_Int package (exponential
  function on integers)
* adapt spark lib, symbols, and recipes to incorporate the Exp_Int package

Fixes #3848
 2020-08-25 11:50:09 +02:00
Martin Stein
f3eaeb08ef ada/spark: all warnings, warn strict, style checks 
* enable all common warnings through default value of CC_ADA_WARN
* treat warnings like errors through default value of CC_ADA_WARN_STRICT
* enable almost all style checks through default value of CC_ADA_WARN_STRICT
* style fixes for aes_cbc_4k
* disable strict warnings and style checks for libsparkcrypto and spark lib

Ref #3848
 2020-08-25 11:50:09 +02:00
Alexander Boettcher
3f97269988 sculpt_manager: add graphical CPU affinity config 
Fixes #3820
 2020-08-25 11:50:09 +02:00
Norman Feske
1aed881313 Remove drivers/gpio/spec directory 
Issue #2190
 2020-08-25 11:50:09 +02:00
Norman Feske
3c78265e66 Remove drivers/framebuffer/spec directory 
Thanks to the unification of build directories, the sub directories
(rpi, sdl, imx53) do not depend on SPEC values.

Issue #2190
 2020-08-25 11:50:09 +02:00
Norman Feske
fcb21732e0 Move drivers/input/spec/ps2 to drivers/ps2 
This simplifies the directory structure.

Issue #2190
 2020-08-25 11:50:09 +02:00
Norman Feske
db0e86fa41 Remove drivers/input/dummy pseudo driver 
With the transition from the input-session to the event-session
interface, the use case for this component has vanished.

Issue #3827
 2020-08-25 11:50:09 +02:00
Norman Feske
2e22498e5a nitpicker: no framebuffer and input by default 
This patch prevents nitpicker from requesting a framebuffer and input
session by default because the regular use of nitpicker relies on the
capture-session and event-session interfaces by now.

For supporting the nested use of nitpicker via the gui_fb component, it
is still possible to enable the traditional behavior by explicitely
setting the 'request_input' and 'request_framebuffer' config attributes
to "yes".

Issue #3812
 2020-08-25 11:50:09 +02:00
Norman Feske
264f695373 Remove transitionary input_event_client component 
With the transition of the input servers to event clients completed,
this helper component is no longer needed.

Issue #3845
Issue #3827
 2020-08-25 11:50:09 +02:00
Norman Feske
a2381c7e4c Turn USB HID drivers into event-session clients 
Issue #3845
 2020-08-25 11:50:09 +02:00
Norman Feske
974118acec Turn PS/2 driver into event-session client 
Issue #3845
 2020-08-25 11:49:45 +02:00
Christian Prochaska
08ef528577 stdcxx: enable thread features 
Issue #2442
 2020-08-25 11:49:45 +02:00
Norman Feske
50b367c076 Remove dde_zircon 
Fixes #3847
 2020-08-25 11:49:44 +02:00
Christian Helmuth
36b6ebc030 libc: process VFS operations in monitor 
Fixes #2635
 2020-08-25 11:49:44 +02:00
Christian Helmuth
a89d61acf2 base: call Io_progress_handler only in signal RPC 
This remove the call to Io_progress_handler::handle_io_progress() from
wait_and_dispatch_one_io_signal() to prevent unexpected nesting
I/O-progress handling in case of custom dispatch loops (e.g., in libc).
The original intention of Io_progress_handler was to inform the
entrypoint just before blocking in the entrypoint loop.

Issue #2635
 2020-08-25 11:49:44 +02:00
Norman Feske
9e42a9ac7e Replace server/acpi_input with app/acpi_event 
Fixes #3827
 2020-08-25 11:49:44 +02:00
Norman Feske
19690193a4 Remove obsolete use of input_session/connection.h 2020-08-25 11:49:44 +02:00
Norman Feske
78047b5bd8 Rename fb_boot_drv to boot_fb_drv 
to be consistent with the naming of the other drivers.
 2020-08-25 11:49:43 +02:00
Norman Feske
9662d89cfb Replace input filter with event filter 
This commit applies the transition from the "Input" session to the "Event"
session to the event-filtering mechansim. The functionality of the
input_filter is now provided by the event_filter. The event filter
requests only one "Event" session as destination for the filter result,
which is usually routed to the nitpicker GUI server. It provides an
"Event" service to which any number of event sources can connect.

The configuration of the filter chain remains almost the same. Only the
declaration of the <input> nodes is no longer needed. Instead, the
configuration must specify <policy> nodes, which define the mapping of
"Event" clients (event sources) to the inputs used in the filter chain.

The patch adjusts all uses of the nitpicker GUI server accordingly such
that the event filter reports events to nitpicker's event service
instead of having nitpicker request an "Input" session. This dissolves
the dependency of nitpicker from input drivers.

Issue #3827
 2020-08-25 11:49:43 +02:00
Norman Feske
bc5b161260 os: event_dump server for printing events 
This component takes over the role of the test-input component.

Issue #3827
 2020-08-25 11:49:43 +02:00
Norman Feske
c502e1d095 input_filter: prepare for refactorization 
This patch brings the input filter into a shape that is easier to
re-mold into an event filter, reversing the client/server roles of
the component.

* The 'Sink &destination' is no longer passed as constructor argument
  to the individual filters but passed as argument to the 'generate'
  method. This way, the final destination does not need to exist at
  the construction time of the filter chain but can be created on
  the fly (clearing the way for using 'Event::Client::with_batch').

* A new 'Source::Filter' interface with the 'apply' method aids the
  cascading of filters during 'generate'. The modules now implement
  the 'Source::Filter::filter_event' interface instead of the
  'Source::Sink::submit_event' interface.

* Since the 'Sink &destination' is no longer a member of the filter
  modules, character-repeat events can no longer be emitted in an
  ad-hoc way. Instead, the character-repeat mechanism now invokes
  a new 'Trigger::trigger_generate' hook that prompts the execution
  of the regular 'generate' mechanism by the main program.

This patch is supposed to leave the semantics of the input filter
unchanged (validated by the input_filter.run script).

Issue #3827
 2020-08-25 11:49:43 +02:00
Norman Feske
6c4ce86a34 event_session: make Session_client::Batch abstract 
This way, there can exist multiple implementations of the interface such
as event filters.

Issue #3812
 2020-08-25 11:49:43 +02:00
Norman Feske
7bac1ec7e5 Adapter for connecting input and event servers 
This is an intermediate helper component for the transition of input
servers into event clients. It works analogously to the
input_event_bridge but it connects servers instead of clients.
In particular, it can act as glue between a traditional input driver
that provides an input service and the event filter that provides an
event service. Thanks to this component, each driver can be migrated to
the event-client interface individually.

Issue #3827
 2020-08-25 11:49:43 +02:00
Norman Feske
7932c76d85 nitpicker: add event service 
Issue #3812
 2020-08-25 11:49:43 +02:00
Norman Feske
66c520cdae Remove stale virtualbox_nic_router.run script 
The run script is not regularly tested and thereby has become
nonoperational since a long time.
 2020-08-25 11:49:43 +02:00
Christian Helmuth
9453287a6b libc: move current working directory into kernel 2020-08-25 11:49:43 +02:00
Christian Helmuth
40e936911f libc: remove unused member from Errno 2020-08-25 11:49:43 +02:00
Christian Helmuth
d9121e50c3 libc: translate file-descriptor exhaustion into EMFILE 
Fixes #3841
 2020-08-25 11:49:43 +02:00
Christian Helmuth
cb15f40028 Remove remnants of libc_lxip 
Issue #2960
 2020-08-25 11:49:42 +02:00
Alexander Boettcher
ce1d0464b8 ssh_terminal: destroy libc fd within user context 
Fixes #3840
 2020-08-25 11:49:42 +02:00
Alexander Boettcher
bed531b604 libc: initialize tv struct in pselect 
Fixes #3839
 2020-08-25 11:49:42 +02:00
Christian Helmuth
257b3b6775 libc: cleanup monitor implementation 
- Explicit types for function and monitor execution results
- Remove pending flag and mutex (pending flag was moved to kernel)
 2020-08-25 11:49:42 +02:00
Christian Helmuth
420ed91480 linux_nic_drv: stop receiver thread on congested queue 
Otherwise, the thread spins on and drops packets.
 2020-08-25 11:49:42 +02:00
Christian Prochaska
c6b17be744 vfs/lwip: clear the packet queue in '~Udp_socket_dir()' 
Fixes #3836
 2020-08-25 11:49:42 +02:00
Christian Helmuth
d266f44ef3 libc: free resources on socket creation failure 
Issue #3766
 2020-08-25 11:49:42 +02:00
Christian Helmuth
5a2ac73b69 vfs/lwip: prevent pbuf leakage on ENOTCONN 
Issue #3766
 2020-08-25 11:49:42 +02:00
Christian Helmuth
7996fc45f3 vfs/lwip: signal ENOTCONN via EOF on remote file 
Issue #3766
 2020-08-25 11:49:42 +02:00
Alexander Boettcher
a03a37b1d9 vfs/lwip: avoid leaking of pbuf objects 
Issue #3766
 2020-08-25 11:42:37 +02:00
Christian Helmuth
76ac94f4f2 vfs/lwip: fix pbuf slab-block size 
Issue #3766
 2020-08-25 11:42:37 +02:00
Alexander Boettcher
f4d3231034 cpu_sampler: handle all quota upgrades 
Beforehand cap_quota upgrades got ignored which leads to non working clients.

Issue #3826
 2020-08-25 11:42:37 +02:00
Alexander Boettcher
b7ffeb51aa cpu_sampler: avoid spinning on unavailable state 
Fixes #3826
 2020-08-25 11:42:37 +02:00
Alexander Boettcher
ed15a46ca4 base: construct child process only once 
Fixes #3821
 2020-08-25 11:42:36 +02:00
Alexander Boettcher
c55a499009 base: remove delayed dispatch from Rpc_entrypoint 
Fixes #3833
 2020-08-25 11:42:36 +02:00
Alexander Boettcher
d1be1281bc os: use whole memory in Packet_allocator 2020-08-25 11:42:36 +02:00
Christian Prochaska
377dadd461 libc_getaddrinfo.run: fix run script 
Fixes #3832
 2020-08-25 11:42:36 +02:00
Christian Prochaska
353baa9251 libc: add pthread alias functions with underscore 
The libc calls pthread functions with underscore internally.

Issue #725
 2020-08-25 11:42:36 +02:00
Alexander Boettcher
60106ac2c8 base: avoid deadlock and page fault 
Fixes #3830
 2020-08-25 11:42:29 +02:00
Christian Helmuth
de7d4a5523 Replace libc_pipe plugin by VFS pipe plugin 2020-07-30 08:49:24 +02:00
Christian Prochaska
9b6cc75f1c tool/seccomp: allow 'restart_syscall' on x86_64 
Fixes #3831
 2020-07-30 08:49:23 +02:00
Christian Helmuth
18b26ff595 vfs/pipe: never report read_ready on pipe/out handle 2020-07-30 08:49:23 +02:00
Christian Helmuth
e44ffa02c9 libc: remove redundant dummies (fstatat, stat) 2020-07-30 08:49:23 +02:00
Christian Helmuth
5122917d62 os: declare Path::max_len() const 2020-07-30 08:49:23 +02:00
Alexander Boettcher
3639a1af80 acpi_drv: avoid out-of-bound access to FADT table 
Fixes #3829
 2020-07-30 08:49:23 +02:00
Alexander Boettcher
851b842033 vbox5-generic: use pthread_cond_timedwait for halt 
use similar implementation as introduced for vbox5-nova

vbox5-nova: avoid Blocking_canceled exception

Related to #3810
 2020-07-30 08:49:23 +02:00
Alexander Boettcher
072a00ba18 vbox5: remove ubuntu 14_04 test 2020-07-30 08:49:23 +02:00
Alexander Boettcher
7e0b66835b vbox5: show serial for ubuntu VMs during autopilot 2020-07-30 08:49:23 +02:00
Alexander Boettcher
73e671893b nova: remove obsolete local vcpu case 
Seoul was the last user of Vcpu_same_pd
 2020-07-30 08:49:23 +02:00
Christian Prochaska
fd682cd470 hw: check descriptor validity in LPAE lookup 
Fixes #3828
 2020-07-30 08:49:23 +02:00
Norman Feske
bfd94d64ba pbxa9_drivers: add blit api 
This is a follow-up fix of "Turn framebuffer drivers into capture
clients".

Fixes #3813
 2020-07-30 08:49:23 +02:00
Piotr Tworek
d8539af412 platform_session: Add missing include. 
The code uses Genode::Attached_dataspace but fails to include header
defining this type.

Fixes #3824
 2020-07-30 08:49:23 +02:00
Norman Feske
cd7a6fc9fe pkg/drivers_interactive-muen: use capture session 
This is follow-up commit to "Turn framebuffer drivers into capture
client".

Issue #3813
 2020-07-30 08:49:23 +02:00
Alexander Boettcher
41380ff769 base: remove Cancelable_lock 
- base/cancelable_lock.h becomes base/lock.h
- all members become private within base/lock.h
- solely Mutex and Blockade are friends to use base/lock.h

Fixes #3819
 2020-07-30 08:49:23 +02:00
Alexander Boettcher
0ed7367c97 vbox5-nova: avoid using semaphore used by Lock 
Issue #3819
 2020-07-20 15:59:47 +02:00
Alexander Boettcher
eab09a2f7c nova: avoid Blocking_canceled exception in timer 
Issue #3819
 2020-07-20 15:59:47 +02:00
Norman Feske
de795b1a6e depot: update recipe hashes 2020-07-13 11:33:53 +02:00
Alexander Boettcher
73546a135a seoul: increase VMM memory for vm_stress* 
The memory was not sufficient if more than 2 vcpus are used. The test starts
up to 4 vCPUs.
 2020-07-13 11:33:15 +02:00
Alexander Boettcher
c888d856ee vbox5*.run: disable in autopilot usb usage 
to avoid to spam the log with usb messages about our usb dongle sending
regularly input events
 2020-07-13 11:33:15 +02:00
Norman Feske
c81af531a3 Turn framebuffer drivers into capture clients 
This patch replaces the use of the "Framebuffer" session interface by
the new "Capture" session interface in all framebuffer drivers. Thanks
to this change, those drivers have become mere clients of the nitpicker
GUI server now, and are no longer critical for the liveliness of the GUI
server.

The patch touches the following areas:

- The actual driver components. The new versions of all drivers have
  been tested on the respective hardware. Generally, the drivers
  have become simpler.

- The drivers_interactive packages for various boards. The drivers
  subsystem no longer provides a "Framebuffer" service but needs a
  valid route to the "Capture" service provided by nitpicker.

- The driver manager of Sculpt OS.

- This patch changes the role of the test-framebuffer component from a
  framebuffer client to a capture server so that drivers (capture clients)
  can be directly connected to the test component without the nitpicker
  server.

- Framebuffer driver no longer support the unbuffered mode.

- The fb_bench.run script is no longer very meaningful because it
  interplays solely with nitpicker, not with the driver directly.

- All run scripts for graphical scenarios and the related depot
  archives got adapted to the change.

Fixes #3813
 2020-07-13 11:33:15 +02:00
Norman Feske
6743669ab8 nitpicker: report no mode smaller than 1x1 pixels 
Issue #3812
 2020-07-13 11:33:14 +02:00
Norman Feske
320387db89 nitpicker: improve GUI client resize handling 
This patch replaces meta-data allocation during the resize handling by a
new 'Resizeable_texture' type that has all meta data preallocated.
It also replaces the use of pointer return values with the
'Resizeable_texture::with_texture' method.

Issue #3812
 2020-07-13 11:33:14 +02:00
Norman Feske
4cad1a87df nitpicker: make framebuffer and input optional 
Nitpicker used to unconditionally request a framebuffer and input
session. With the transition to the new capture/event session
interfaces, this built-in policy does no longer suffice.

This patch introduces the attributes 'request_framebuffer="yes"' and
'request_input="yes"' (with the default values shown) to nitpicker's
<config> node. If setting those attributes to "no", nitpicker won't
request a "Framebuffer" and "Input" session respectively.

Issue #3812
 2020-07-13 11:33:14 +02:00
Norman Feske
22cb6dded7 nitpicker: add capture service 
Issue #3812
 2020-07-13 11:33:14 +02:00
Norman Feske
537b317273 nitpicker: handle failed try_alloc_texture 
Fixes #3782
 2020-07-13 11:33:14 +02:00
Norman Feske
795a817a33 nitpicker: remove dirty_rect state from view stack 
In the presence of potentially multiple output back ends, this
dirty_rect state must be maintained individually per back end. Instead
of storing the dirty_rect as view-stack member, the view stack now calls
a new 'Damage::mark_as_damaged' interface, which allows nitpicker to
propagate this information to multiple back ends. Unfortunately, the
patch must remove the per-view dirty_rect state.

Issue #3812
 2020-07-13 11:33:14 +02:00
Norman Feske
067a7ad7e9 nitpicker: internal cleanup 
This patch simplifies the internal naming of the nitpicker GUI server as
a preparatory step for adding support for the capture session interface.

Issue #3812
 2020-07-13 11:33:14 +02:00
Norman Feske
3863de9589 Input-event-bridging service 
The sole purpose of this service is the migration path from the use of
input session interface to the event session interface.

Issue #3812
 2020-07-13 11:33:14 +02:00
Norman Feske
443d3c98dd Event session interface 
Issue #3812
 2020-07-13 11:33:13 +02:00
Norman Feske
602def9bdd Simple example for using a capture connection 
Issue #3812
 2020-07-13 11:33:13 +02:00
Norman Feske
3d1d1b439d Capture session interface 
Issue #3812
 2020-07-13 11:33:13 +02:00
Josef Söntgen
49e907e5f6 libc: remove trailing slash in lstat and stat 
Fixes #3814.
 2020-07-13 11:33:13 +02:00
Alexander Boettcher
c4a8c6798b top: react on too small trace argument buffer size 
Issue #3811
 2020-07-13 11:33:13 +02:00
Alexander Boettcher
21e48a8e12 trace: extend for_each_subject_infos return value 
Fixes #3811
 2020-07-13 11:33:13 +02:00
Norman Feske
a7d170adda os: add blit/painter.h 
This utility eases the application of the 'blit' function when using the
'Texture' and 'Surface' types.
 2020-07-13 11:33:13 +02:00
Norman Feske
91b71f7a9f run: add drivers_interactive for panda and arndale 2020-07-13 11:33:13 +02:00
Alexander Boettcher
641679f7e7 vbox5-nova: avoid Blocking_canceled exception 
using pthread primitives to implement block/wakeup of EMT thread

Fixes #3810
 2020-07-13 11:33:13 +02:00
Christian Helmuth
ed4594c76b Fix success/failure detection in cpu_bench testsuite 
Issue #3785
 2020-07-13 11:33:12 +02:00
Alexander Boettcher
997a77b3de foc: use Mutex 
Issue #3809
 2020-07-13 11:33:12 +02:00
Alexander Boettcher
6fa4307005 foc: keep core internal pager state separately 
beforehand the Lock object and more was accessible
via FOC native cpu thread RPC

Fixes #3809
 2020-07-13 11:33:12 +02:00
Alexander Boettcher
f3efbe50bb base: remove deprecated cancel_blocking() support 
for threads.

Fixes #3806
 2020-07-13 11:33:12 +02:00
Alexander Boettcher
ce6f4dffe5 linux: use Mutex 
Fixes #3807
 2020-07-13 11:33:12 +02:00
Norman Feske
a0b0892df3 UEFI/boot_fb: constrain Qemu to 1920x1080 
Fixes #3808
 2020-07-13 11:33:12 +02:00
Christian Helmuth
e163055f6a ram fs: Release node mutex on unlink 
This is needed if an open file is unlinked.

Fixes #3800
 2020-07-13 11:33:12 +02:00
Alexander Boettcher
7acc34b48b dde_zircon: use Mutex 
Fixes #3805
 2020-07-13 11:33:12 +02:00
Alexander Boettcher
a9827c662e dde_ipxe: use Mutex 
Issue #3805
 2020-07-13 11:33:12 +02:00
Alexander Boettcher
9a8a42e819 dde_rump: use Mutex/Blockade 
Issue #3805
 2020-07-13 11:33:12 +02:00
Alexander Boettcher
5516dbcb1f dde_linux: use Mutex/Blockade 
Issue #3805
 2020-07-13 11:33:11 +02:00
Alexander Boettcher
b7cdb5840a dde_bsd: remove unused Lock 
Issue #3805
 2020-07-06 14:43:05 +02:00
Alexander Boettcher
61d798c629 launchpad: use Mutex 
Fixes #3804
 2020-07-06 14:43:05 +02:00
Alexander Boettcher
973fbc98be nitlog: use Mutex 
Issue #3804
 2020-07-06 14:43:05 +02:00
Alexander Boettcher
3d634df34d gems: remove report_rom_slave.h 
Fixes #3803
 2020-07-06 14:43:04 +02:00
Alexander Boettcher
ad284491e6 mixer_gui_qt: use Blockade 
Issue #3803
 2020-07-06 14:43:04 +02:00
Alexander Boettcher
f946de4450 tcp_terminal: use Mutex 
Issue #3803
 2020-07-06 14:43:04 +02:00
Alexander Boettcher
377c6ed0ce ssh_terminal: use Mutex 
Isse #3803
 2020-07-06 14:43:04 +02:00
Alexander Boettcher
1609d8a92f gdb_server: use Mutex 
Fixes #3802
 2020-07-06 14:43:04 +02:00
Alexander Boettcher
169cad6059 seoul: use Mutex 
Issue #3802
 2020-07-06 14:43:04 +02:00
Alexander Boettcher
bba4790002 virtualbox5: use Mutex/Blockade 
Issue #3802
 2020-07-06 14:43:04 +02:00
Alexander Boettcher
c943e26913 lib/drm: use Blockade 
Fixes #3801
 2020-07-06 14:43:04 +02:00
Alexander Boettcher
7d1fd0b0a3 lwip: use Mutex 
Issue #3801
 2020-07-06 14:43:04 +02:00
Alexander Boettcher
6d653b3c0f qemu-usb: use Mutex 
Issue #3801
 2020-07-06 14:43:04 +02:00
Alexander Boettcher
5936d00b5e acpica: use Mutex 
Issue #3801
 2020-07-06 14:43:04 +02:00
Christian Prochaska
5af4552511 qt5: build Qt libraries and applications with qmake 
Fixes #3795
 2020-07-03 13:48:55 +02:00
Norman Feske
0310c733d5 base-linux: let seccomp permit 'read' 
This is needed for using the 'wait_for_continue' debug mechanism.

Fixes #3798
 2020-07-03 11:58:31 +02:00
Christian Helmuth
0e6c32f75e libc: use Mutex/Blockade 
Fixes #3796
 2020-07-03 11:16:20 +02:00
Christian Helmuth
93ab972ddc tool/seccomp: allow sigreturn on x86 
Fixes #3799
 2020-07-03 11:15:33 +02:00
Norman Feske
35c3acdf05 depot: update recipe hashes 2020-07-03 11:13:59 +02:00
Alexander Boettcher
103236fdca nitpicker: fix memory accounting during realloc 
Fixes #3794
 2020-07-03 11:11:24 +02:00
Christian Prochaska
62848b1a68 mk: set soname in abi libraries 
Fixes #3793
 2020-07-03 11:11:23 +02:00
Alexander Boettcher
64ef651d4d libc: handle MSG_WAITALL in socket_fs_plugin 
Fixes #3774
 2020-07-03 11:11:23 +02:00
Emery Hemingway
1a620acc17 libc: add newlocale and freelocale dummies 
Fix #3777
 2020-07-03 11:11:23 +02:00
Christian Prochaska
6e71208db3 libc: fix timeout calculation in 'ppoll()' 
Fixes #3791
 2020-06-30 09:15:53 +02:00
Christian Helmuth
8e205e0324 sel4: assignment of BOARD as effective override 
Issue #3792
 2020-06-30 09:11:57 +02:00
Christian Prochaska
9578fadae2 libc: handle nullptr timeout in 'ppoll()' 
Fixes #3790
 2020-06-29 16:59:35 +02:00
Alexander Boettcher
1ec823bf5e libc: support manual pthread configuration 
Fixes #3787
 2020-06-29 16:56:47 +02:00
Alexander Boettcher
2471410fe5 libc: provide number of CPUs for sysctl&sysconf 
Fixes #3786
 2020-06-29 16:55:56 +02:00
Alexander Boettcher
3fb5ae4fdc vm_stress: fix resource request by seoul-debian32 
+ increase VM RAM to same size as used by vbox5
 2020-06-29 16:54:43 +02:00
Norman Feske
4450b37ff5 depot: update recipe hashes 2020-06-29 14:25:28 +02:00
Alexander Boettcher
309597bbda timeout fw: be robust against past _timestamp() 
_timestamp() returns CPU local values which may not be in sync with _ts
taken from another CPU. Be robust and don't produce wraparound/negative
timeout values.

Issue #3657
 2020-06-29 14:25:27 +02:00
Norman Feske
7dbf836217 Move stubby to genode-world 
Fixes #3789
 2020-06-29 14:25:27 +02:00
Norman Feske
5249f3358f Move Solo5 to genode-world 
Fixes #3761
 2020-06-29 14:25:27 +02:00
Stefan Kalkowski
1f91fd3f7d hw: scale imx8q_evk cpu to 1.5 GHz 2020-06-29 14:25:27 +02:00
Alexander Boettcher
01676717e2 tool: remove outdated tianocore image 
It is known to work up to Qemu 2.9 solely.

Instead use ovmf package provided by Linux distributions.

Issue #3784
 2020-06-29 14:25:27 +02:00
Stefan Kalkowski
1cfb1af56e os: automate cpu_bench testsuite 
* Differentiate in between different architectures with assembler routines
  for correct measures
* Automate first step measuring of 10G bogomips across different hardware

Fix #3785
 2020-06-29 14:25:27 +02:00
Norman Feske
ef741ef80d Change pixel format to 32 bits per pixel 
Until now, Genode's framebuffer session interface was based on the
RGB565 pixel format. This patch changes the pixel format to 32-bit
XRGB where the X part is ignored. It adapts all graphical applications
and device drivers accordingly.

The patch also adjusts the users of the drivers_interactive packages,
assigning 64 MiB RAM and 1500 caps to the drivers subsystem, which is
sufficient for covering high resolutions at 32 bits per pixel and to
accommodate multi-component USB HID input stacks.

Fixes #3784
 2020-06-29 14:22:29 +02:00
Alexander Boettcher
6119e03081 grub2: avoid hardcoding boot disc 
see alex-ab/g2fg#1
 2020-06-29 14:22:28 +02:00
Josef Söntgen
6d945e6a61 os: remove rom_block server component 
This component is superceeded by vfs_block.

Issue #3781.
 2020-06-29 14:22:28 +02:00
Josef Söntgen
6fc7ed55cf Replace 'rom_block' with 'vfs_block' 
Issue #3781.
 2020-06-29 14:22:28 +02:00
Josef Söntgen
8b590e2330 os: remove ram_block server component 
This component is superceeded by vfs_block.

Issue #3781.
 2020-06-29 14:22:28 +02:00
Josef Söntgen
ec711b008e Replace 'ram_block' with 'vfs_block' 
Issue #3781.
 2020-06-29 14:22:28 +02:00
Josef Söntgen
e5b00d89fb depot_autopilot: test case for vfs_block 
Issue #3781.
 2020-06-22 09:50:51 +02:00
Josef Söntgen
be55f080e4 os: add test-vfs_block pkg recipe 
Issue #3781.
 2020-06-22 09:50:51 +02:00
Josef Söntgen
e1aab829ca os: Block session server backed by VFS library 
The 'vfs_block' component will give access to a VFS file through a
Block session. For more detailed information please look at its
README.

(On a technical note, the server currently only allows for one
active session and has only one pending back end request but can
easily be extended in the future.)

Fixes #3781.
 2020-06-22 09:50:51 +02:00
Josef Söntgen
e56dd15a4b vfs: extend <zero> fs to limit amount of zeros 
Setting the 'size' attribute will restrict a reader to consume
0s up to the specified amount per VFS handle. The attribute is
parsed as 'Number_of_bytes', which will limit the amount to the
maximum of the addressable memory on the platform.

This addition comes in handy when needing to provide a synthetic
empty file with a fixed size.

Issue #3781.
 2020-06-22 09:50:51 +02:00
Josef Söntgen
103ae9df4a block_tester: limit batching in sequential test 
So far the condition whether to spawn a new job or not depended on
the amount of data already processed. This could lead to spawning
more jobs than necessary if batching is used and in return could
result in creating invalid requests in case the tested block session
is not large enough.

In addition to checking the amount of data the test now stores the
number of the last block and checks if the current request is in
range. This properly limits the total amount of requests.

Issue #3781.
 2020-06-22 09:50:51 +02:00
Josef Söntgen
b5f0c07eb3 os: add block_tester src recipe 
Issue #3781.
 2020-06-22 09:50:51 +02:00
Norman Feske
eb3a81a874 depot: update recipe hashes 2020-06-22 09:39:41 +02:00
Martin Stein
1e96510815 server/vfs: set seek offset before executing write 
The seek offset has to be updated on partial writes according to the
current write position. Otherwise writes will override the content
at the initial offset.

Fixes #3780.
 2020-06-22 09:39:40 +02:00
Stefan Kalkowski
04d8c859d8 hw: enable branch predictor on rpi 
* Z-bit was never enabled for rpi1
* Write buffer, and TCM bits are deprecated in this processor generation
  so do not use them

Ref #3247
 2020-06-22 09:39:40 +02:00
Stefan Kalkowski
b11da67679 usb_hid: introduce cpu quota 
Ref #3247
 2020-06-22 09:39:40 +02:00
Norman Feske
c8322ffd2a decorator: fix cap-upgrade handling 
This commit fixes the 'Out_of_caps' handling. The previous version
mistakenly upgraded the RAM instead the caps as response. The problem
could be triggered by the 'decorator_stress.run' script.
 2020-06-22 09:39:40 +02:00
Norman Feske
f63713694c api/blit: package arch-specific optimizations 
The blit API archive missed the CPU-architecture-specific variants of
the blit library such that all depot binaries worked with the generic
(slow) fallback implementation. This patch adds the missing pieces to
the blit API archive.
 2020-06-22 09:39:40 +02:00
Norman Feske
7a97cd70aa Rename nit_fader to gui_fader 
Issue #3778
 2020-06-22 09:39:40 +02:00
Norman Feske
48b4891f6e Rename nit_fb to gui_fb 
Issue #3778
 2020-06-22 09:39:40 +02:00
Norman Feske
5d40c0c1ce Rename 'Nitpicker' namespace to 'Gui' 
Issue #3778
 2020-06-22 09:39:40 +02:00
Norman Feske
e8f5706382 Rename <nitpicker> in runtime files to <gui> 
Issue #3778
 2020-06-22 09:39:39 +02:00
Norman Feske
98f39c698f Rename "Nitpicker" service name to "Gui" 
Issue #3778
 2020-06-22 09:39:39 +02:00
Norman Feske
c6eda9bd80 Rename include/nitpicker_session to gui_session 
This patch also renames the depot api archive accordingly.

Issue #3778
 2020-06-22 09:39:39 +02:00
Alexander Boettcher
87b08d6c7f libc: support pthread placement configuration 
Up to now all pthreads get placed on one CPU.

The patch adds support to evaluate a libc pthread configuration specifying
a placement strategy.

The default strategy is now to use all available CPUs as imposed by Genode's
affinity configuration for the pthread application.

The old behaviour, putting all pthreads on one single cpu can be still
configured by:

<libc>
 <pthread placement="single-cpu"/>
 ...
</libc>

Fixes #3775
 2020-06-22 09:39:39 +02:00
Alexander Boettcher
5f7e670ebc release_notes: fix Sculpt ARM 64 topic branch link 2020-06-22 09:39:39 +02:00
Josef Söntgen
e8fec3eed6 terminal_log: remove session size check 
The amount of RAM required by the session object is already accounted
for by the Root_component.

Fixes #3776.
 2020-06-22 09:39:38 +02:00
Emery Hemingway
dc8b4eeb40 cached_fs_rom: diagnosis of delivered ROM sessions 
To enable logging of ROM session deliveries, set the "diag" on
session routes:

<route>
  <service name="ROM" label_prefix="/bin/">
    <child name="cached_fs_rom" diag="yes"/>
  </service>
  …
<route>

Fix #3772
 2020-06-22 09:39:38 +02:00
Christian Helmuth
4f87fbd5ae Support linux and mod_openssl in lighttpd test 2020-06-22 09:39:38 +02:00
Christian Helmuth
df3f7dc1bc Support linux in manual fetchurl test 2020-06-22 09:39:38 +02:00
Christian Helmuth
8e85d889f1 libc: generate osreldate.h 2020-06-22 09:39:38 +02:00
Norman Feske
f687b0f3b9 News item for Sculpt 20.05 2020-06-22 09:39:38 +02:00
Christian Helmuth
5b87f68900 version: 20.05 2020-05-28 11:59:30 +02:00
Norman Feske
7221199f74 Release notes for version 20.05 2020-05-28 11:50:55 +02:00
Christian Helmuth
6006051fb9 depot: update recipe hashes 2020-05-27 11:56:47 +02:00
Christian Helmuth
63048fb89f doc: replace http by https in links where possible 
This also fixes some mixed content pages on genode.org and, thus,
removes the ugly browser warning, e.g., on
https://genode.org/documentation/release-notes/17.05.
 2020-05-27 11:56:47 +02:00
Norman Feske
a90aa78c6e doc: update components.txt 2020-05-27 11:56:46 +02:00
Stefan Kalkowski
70acd4b2d5 os: pl050, pl11x, lan9118 use ARM platform driver 
Make the framebuffer driver for pl11x chipsets,
the ps2 input driver for pl050, and the lan9116 NIC driver independent from
the pbxa9 board by using the newly introduced common ARM platform driver API.

Ref #3299
 2020-05-27 11:56:46 +02:00
Stefan Kalkowski
1a80f166c5 os: introduce common platform driver for ARM 
Ref #3299
 2020-05-27 11:56:46 +02:00
Stefan Kalkowski
935dcf8b18 usb_drv: include specific rpi platform_session API 
The rpi_usb_drv uses the rpi-specific variant of the platform_session API.
Therefore, it has to include it excplicitly.

Ref #3299
 2020-05-27 11:56:46 +02:00
Stefan Kalkowski
b915b0adc4 Annotate irq session interface with RAM_QUOTA 
Ref #3299
 2020-05-27 11:56:46 +02:00
Stefan Kalkowski
6e6b671a66 Annotate io_mem session interface with RAM_QUOTA 
Ref #3299
 2020-05-27 11:56:46 +02:00
Stefan Kalkowski
e1333c9421 os: refactor i.MX53 interactive drivers (ref #3299) 
* Remove input driver specific to i.MX53 tablet board from QSB driver pkg
* Move GPIO settings for QSB LVDS backlight out of framebuffer driver into
  GPIO driver config
* Move PWM driver functionality out of framebuffer driver
* Make framebuffer driver configureable, and less dependent on
  platform driver i.MX53 specifics
 2020-05-27 11:56:46 +02:00
Norman Feske
ab8ef5750d doc: tweaks for updated Genode Foundations book 2020-05-27 11:56:46 +02:00
Norman Feske
7c20ba84e4 tool/parse_cxx: consider __attribute__((...)) 
This change is needed to parse the __attribute__((deprecated))
annotations present in 'util/xml_node.h'.
 2020-05-27 11:56:46 +02:00
Stefan Thöni
78497c03ca base-linux: enabled seccomp 
base-linux uses seccomp to reduce the available system calls
to the minimum set needed to run base-linux. There are still
some syscalls that allow accessing global state which should
be further reduced.

The combination of seccomp and socket descriptor caps should
provide an intermediate level of security for base-linux
thereby enabling base-linux as a migration path from using
the Linux kernel to the use of microkernel-based Genode.

Fixes #3581
 2020-05-27 11:56:46 +02:00
Norman Feske
128ba65109 doc: fix autopilot command in release notes 19.08 
Fixes #3764
 2020-05-27 11:56:46 +02:00
Alexander Boettcher
f80b4b9fd9 vm_stress.run: update numbers 
due to script changes to stabilize/minimize variants in runs

Issue #3683
 2020-05-27 11:56:46 +02:00
Pirmin Duss
7f6b6d7e13 core-linux: make RAM quota configurable 
Add the option to configure the RAM quota for core on base-linux.
The environment variable GENODE_RAM_QUOTA if it exists is interpreted
as number of bytes to provide to the first init.

Issue #3762
 2020-05-27 11:56:46 +02:00
Pirmin Duss
51104454aa depot_deploy: copy all attributes of <report> node 
This allows the configuration of all report attributes of the <report>
node for the sub init instead of only `delay_ms`.

Issue #3760
 2020-05-27 11:56:46 +02:00
Christian Prochaska
2f47bbde30 qt5: use VFS pipe plugin 
Fixes #3759
 2020-05-27 11:56:46 +02:00
Norman Feske
b078224753 Replace Genode::strncpy by Genode::copy_cstring 
- Since Genode::strncpy is not 100% compatible with the POSIX
  strncpy function, better use a distinct name.

- Remove bogus return value from the function, easing the potential
  enforcement of mandatory return-value checks later.

Fixes #3752
 2020-05-27 11:56:45 +02:00
Norman Feske
0f27d139bd depot: update recipe hashes 2020-05-18 10:16:59 +02:00
Norman Feske
77a875735a fs_query.run: adjust quota for seL4 2020-05-18 10:16:15 +02:00
Christian Helmuth
32753c3f69 Use HTTPS in fetchurl test 
HTTPS is standard today and HTTP the exception.
 2020-05-18 10:16:15 +02:00
Christian Helmuth
7d2eb71094 openssl: update to current version 1.0.2u 2020-05-18 10:16:15 +02:00
Norman Feske
0de54cddaa gdb_monitor: hide exceptions during create_thread 
This patch is a workaround for the missing implementation of
'Pd_session::transfer_quota' interface by the GDB monitor's PD service.
The missing implementation becomes problematic with the changes of #3750
that enabled the cap-quota accounting for core's CPU service.

In regular scenarios without the GDB monitor, the client of
'Cpu_session::create_thread' deals with Out_of_caps or Out_of_ram by
upgrading the CPU session's cap and RAM quotas. This, in turn, results
in a sequence of 'transfer_quota' operations at the parent.

Since GDB monitor implements a custom PD service, these 'transfer_quota'
calls try to transfer quota between sessions provided by core and those
provided by the GDB monitor. This does of course not work. To fix this
issue, the GDB monitor needs a major overhaul. This patch side-steps
the problem by handing Out_of_caps and Out_of_ram from the debuging
target.
 2020-05-18 10:16:15 +02:00
Norman Feske
4002653334 qt5: fix audio driver support on Linux 
The lx_hybrid audio driver must be started with the 'ld="no"' argument.
 2020-05-18 10:16:15 +02:00
Emery Hemingway
adb48b5c9e Libc: MSG_PEEK support 
As discovered by Johannes Kliemann, peeking at buffered socket data
using 'recv' and 'MSG_PEEK' is not supported. Read a "peek" control file
from the socket directory to attempt to peek into buffers at the
socket_fs. Support for every feature of POSIX sockets cannot be
expected, but this one is trivial to implement.

Fix #2875
 2020-05-18 10:16:15 +02:00
Emery Hemingway
7f928a6573 LxIP "peek" file 
Add a control file for peeking at socket buffers.

Ref #2875
 2020-05-18 10:16:15 +02:00
Josef Söntgen
f1427fc02a dde_bsd: remove configration attributes 
The driver always opens the audio device in duplex mode, i.e.,
playback and recording. Setting the 'playback' or 'recording' attribute
only influnces the service announcement. Due to changes made in a more
recent OpenBSD release recording must be set enabled explicitly anyway.
Since we already provide the interface that mirrors the one used by
OpenBSD 1:1 in the configuration use that and the remove the additional
config attributes.

Fixes #3757.
 2020-05-18 10:16:14 +02:00
Josef Söntgen
8ce798abed dde_bsd: document audio driver configuration 
So far the configuration was only document in the commit message that
introduce the change.

Issue #3757.
 2020-05-18 10:16:14 +02:00
Norman Feske
dd0c1575f5 Xml_node: mark deprecated methods 
Fixes #3755
 2020-05-18 10:16:14 +02:00
Norman Feske
4b9c1d8f2b solo5: tolerate use of deprecated Xml_node API 
This patch is an interim fix to allow building solo5 despite of the use
of deprecated parts of the Xml_node API.

Issue #3755
 2020-05-18 10:16:14 +02:00
Norman Feske
be65c4acd2 Avoid use of deprecated Xml_node methods 
Issue #3755
 2020-05-18 10:16:14 +02:00
Norman Feske
d22b95ded3 test-fs_rom_update_ram: increase timeout for foc 
The timeout triggered on foc/x86_32 in Qemu.
 2020-05-18 10:16:13 +02:00
Christian Helmuth
c11d9b7b5c Remove false API dependencies from libc/posix components 
Fixes #3720
 2020-05-18 10:16:13 +02:00
Christian Helmuth
42fddf8390 Cleanup shared-object support mechanics 
The former ldso-startup static library (now called ldso_so_support) is
used to spice each shared object/library with local support code for the
dynamic linker (execution of static constructors and ARM-EABI).
Therefore, the library must be statically linked to each dynamic
library.

As a result recipes for dynamic libraries must always depend on the "so"
API, which makes ldso_so_support.mk and so_support.c available
independent of "base". Additionally, ldso_so_support is also provided in
the libc API to cut the dependency early for libc/posix libraries.

Issue #3720
 2020-05-18 10:16:13 +02:00
Christian Helmuth
589b416ca8 Remove misleading "pipe" utility and test 
If a "cat" tool as simple as the removed one is needed, it should be
implemented with Goa, only depend on libc/posix (*not* base), and be
named "simple_cat". Up to today, the misleading naming of the removed
tool, tricked me into believing "test-pipe" tests POSIX pipes, which it
did not!
 2020-05-18 10:16:13 +02:00
Christian Helmuth
4738d77c88 libc: fix CC_C_OPT in libm 2020-05-18 10:16:13 +02:00
Norman Feske
64bc008c3a core: fix inconsistent state after failed 'trace' 
This patch fixes the handling of the corner case where the allocation of
a trace buffer throws 'Out_of_caps' or 'Out_of_ram'. Under this
circumstance, the '_buffer' would still be flagged with the 'size',
which prevented any subsequent allocation attempt. This patch fixes the
problem by initializing the 'size' after the potentially throwing
allocation.

The problem triggered with the test-trace_logger after the accounting of
core's TRACE service (replacing the 'Allocator_guard' by
'Constrained_ram_allocator') became more accurate.

Related to issue #3750
 2020-05-18 10:16:13 +02:00
Christian Helmuth
236e02a2dd linux: remove initramfs from base-linux src archive 2020-05-18 10:16:13 +02:00
Norman Feske
cb5b688eb9 Remove <configfile> feature from init 
Fixes #3753
 2020-05-18 10:16:13 +02:00
Norman Feske
c39a342fe5 Remove use of <configfile> feature of init 
Issue #3753
 2020-05-18 10:16:13 +02:00
Norman Feske
07502ce6bb Remove Python2 
Fixes #3751
 2020-05-18 10:16:13 +02:00
Norman Feske
48b2456845 util/token.h: fix possible out-of-bounds read 
The 'WHITESPACE' case of the _calc_len method wrongly accessed the
character before checking upper bound of the token. The problem is fixed
by switching the order of both conditions.

Fixes #3756
 2020-05-18 10:16:12 +02:00
Christian Helmuth
894c7411e7 Remove obsolete vmm_utils test 2020-05-18 10:16:12 +02:00
Sebastian Sumpf
ea78d85d35 ahci: SYNC with empty NCQ queue only 
Fixes #3746
 2020-05-18 10:16:12 +02:00
Norman Feske
04aeaa25e5 Remove dead code 2020-05-18 10:16:12 +02:00
Norman Feske
9d67f9fc8e Remove Allocator_guard 
This patch removes old 'Allocator_guard' utility and replaces its use
with the modern 'Constrained_ram_allocator'.

The adjustment of core in this respect has the side effect of a more
accurate capability accounting in core's CPU, TRACE, and RM services.
In particular, the dataspace capabilities needed for core-internal
allocations via the 'Sliced_heap' are accounted to the client now.
The same goes for nitpicker and nic_dump as other former users of the
allocator guard. Hence, the patch also touches code at the client and
server sides related to these services.

The only remaining user of the 'Allocator_guard' is the Intel GPU
driver. As the adaptation of this component would be too invasive
without testing, this patch leaves this component unchanged by keeping a
copy of the 'allocator_guard.h' locally at the component.

Fixes #3750
 2020-05-18 10:16:12 +02:00
Christian Prochaska
1cc7277996 qt5: don't build webkit-related libraries for arm_v8a 
Fixes #3747
 2020-05-05 13:53:09 +02:00
Norman Feske
a9f0e47ea3 Remove return value of Log_session::write 
Fixes #3749
 2020-05-05 13:51:05 +02:00
Norman Feske
bbc21cf063 util/bit_array.h: remove use of memset and memcpy 
This makes the code less dependent on functions considered unsafe.

Fixes #3748
 2020-05-05 13:30:15 +02:00
Christian Helmuth
f722aa3325 os: src recipe for lx_fs 2020-05-04 15:59:29 +02:00
Norman Feske
eb0a33302a libc: strip trailing '/' for directory operations 
Fixes #2686
 2020-05-04 15:28:47 +02:00
Christian Prochaska
151828752c vfs: handle read error in fs file system 
Issue #3742
 2020-04-28 11:43:12 +02:00
Christian Prochaska
6a8fb48c13 vfs server: keep ack packet intact on failure 
Issue #3742
 2020-04-28 11:42:52 +02:00
Christian Prochaska
07b87f6f1f pthread: initialize static condition variables 
Fixes #3741
 2020-04-28 11:42:16 +02:00
Christian Prochaska
905b0c4aef libc: remove log2.patch 
Fixes #3740
 2020-04-28 11:41:38 +02:00
Christian Prochaska
6505ce47ae libc vfs plugin: report non-zero link count in 'stat()' 
Fixes #3739
 2020-04-28 11:40:58 +02:00
Christian Prochaska
31d7b3eb97 vfs_lwip: guard 'blocked_handles' Fifo 
Fixes #3738
 2020-04-28 11:37:56 +02:00
Alexander Boettcher
80d89c20fd top: use batched subject_infos of trace session 
Issue #3610
 2020-04-28 08:58:03 +02:00
Alexander Boettcher
1a94338389 trace: support to request subject infos batched 
Optimize requesting the Subject_infos from O(n) to O(1) RPC call.

Issue #3610
 2020-04-28 08:57:57 +02:00
Norman Feske
dd899fde29 depot: update recipe hashes 2020-04-24 14:37:57 +02:00
Norman Feske
1bf796d69a Remove Rust support 
Fixes #3488
 2020-04-24 14:37:57 +02:00
Norman Feske
c12d76686e Move OpenVPN to genode-world repository 
Fixes #3737
 2020-04-24 14:37:57 +02:00
Stefan Kalkowski
199821a247 os: re-enable drivers_interactive for i.MX53 
* Fix IPU hardware structure layout
* Turn Signal_handler into Io_signal_handler
* Eliminate the faulty msleep in I2C initalization sequence
* Fix routes and resources for tablet board (i.MX53 SMD)

Fix #3735
 2020-04-24 14:37:57 +02:00
Norman Feske
840f383e46 Remove input_merger component 
Fixes #3736
 2020-04-24 14:37:57 +02:00
Norman Feske
1459085a4d mk: prevent duplicates in REPOSITORIES definition 
Fixes #3731
 2020-04-24 14:37:56 +02:00
Norman Feske
6f6340644b Hide binary_ready_hook_for_platform from public 
The hook is meant to be internal to the framework. So better keep its
declaration in the framework-internal globals.h header.

Issue #3581
 2020-04-24 14:37:47 +02:00
Norman Feske
b134867f31 Remove Rpc_entrypoint::Native_context 
This patch largely reverts the commit "base: lay groundwork for
base-linux caps change" because the use of 'epoll' instead of 'select'
alleviated the need to allocate large FD sets, which motivated the
introduction of the 'Native_context' hook.

Related to issue #3581
 2020-04-21 16:50:37 +02:00
Norman Feske
01bf32b998 Remove ram_fs server 
Fixes #3734
 2020-04-21 16:27:07 +02:00
Norman Feske
f14cc2edab Replace use of ram_fs by VFS server 
Issue #3734
 2020-04-21 16:24:37 +02:00
Norman Feske
f0c4fc1e22 sandbox: defer requests for upcoming server 
The sandbox library supports the forwarding of session requests from the
outside to one of the hosted children according to a policy. This patch
introduces the distinction between two cases, which previously triggered
the denial of the session request.

- There exists no matching policy for the requested session
- There exists a matching policy but the referred server child
  does not exist (yet)

Whereas the proper response to the first case is the denial of the
request, the second case can occur in situation where a dynamic init is
used to implement a staged startup, for example via the deploy
mechanism. In such cases, a policy may exist as a static rule while the
server has not been started yet. This patch changes the behavior such
that such requests are stalled.

The patch is accompanied with test cases for exercising both situations.

Fixes #3733
 2020-04-21 14:52:29 +02:00
Alexander Boettcher
f891f4c963 vm_stress: change output message for ci scripts 
so that it detect it correctly as not available instead of a failure on
normal work days

Issue #3683
 2020-04-20 12:36:00 +02:00
Stefan Kalkowski
9f28f4f803 hw: avoid BASE_DIR relative path definitions 
Fix #3728
 2020-04-17 12:55:13 +02:00
Stefan Kalkowski
0e49336b96 Retire Exynos 5 support (fix #3725) 2020-04-17 12:53:57 +02:00
Stefan Kalkowski
941e918b46 vmm: unify armv7/v8 virtualization 
Fix #3638
 2020-04-17 12:52:02 +02:00
Stefan Kalkowski
74e75d7fbc hw: enable virtualization support for virt_qemu 
Ref #3638
 2020-04-17 12:51:53 +02:00
Stefan Kalkowski
58db6542f8 hw: check for security extension support in GICv2 
Ref #3638
 2020-04-17 12:51:44 +02:00
Tomasz Gajewski
fca2a05adf trace: extend test for log_output 
Issue #3714
 2020-04-17 12:48:17 +02:00
Tomasz Gajewski
4639978b3a trace_logger: avoid output of empty lines 
It avoids output of empty lines in output buffer if traced messages end
with \n (what is common with messages logged from dde_linux).

Issue #3714
 2020-04-17 12:48:13 +02:00
Tomasz Gajewski
870d348d77 trace: redirect logs to trace based on policy 
If trace is enabled for component than an attempt to put message into
trace buffer is performed using log_output policy. If it succeeds than
message is not put to logs using log service.

Fixes #3714
 2020-04-17 12:47:48 +02:00
Tomasz Gajewski
db8ec81e9f trace: log_output policy that redirects logs to trace 
Issue #3714
 2020-04-17 12:47:35 +02:00
Tomasz Gajewski
1d9a2dce94 trace: extend policy with log_output 
Issue #3714
 2020-04-17 12:47:19 +02:00
Roman Iten
12ea494477 vfs/File_content: fix end condition in for_each_line 
Fixes #3729
 2020-04-17 12:45:10 +02:00
Christian Helmuth
b60b591d06 depot: update recipe hashes 2020-04-17 12:40:13 +02:00
Christian Helmuth
9fbc68bda1 libports: icu library moved to GitHub 2020-04-17 12:40:13 +02:00
Norman Feske
132569d12b base-linux: socket descriptor caps for RPC 
On Linux, Genode used to represent each RPC object by a socket
descriptor of the receiving thread (entrypoint) and a globally-unique
value that identifies the object. Because the latter was transferred as
plain message payload, clients had to be trusted to not forge the
values. For this reason, Linux could not be considered as a productive
Genode base platform but remained merely a development vehicle.

This patch changes the RPC mechanism such that each RPC object is
represented by a dedicated socket pair. Entrypoints wait on a set of
the local ends of the socket pairs of all RPC objects managed by the
respective entrypoint. The epoll kernel interface is used as the
underlying mechanism to wait for a set of socket descriptors at the
server side.

When delegating a capability, the remote end of the socket pair is
transferred to the recipient along with a plaintext copy of the
socket-descriptor value of the local end. The latter value serves as a
hint for re-identifiying a capability whenever it is delegated back to
its origin. Note that the client is not trusted to preserve this
information. The integrity of the hint value is protected by comparing
the inode values of incoming and already present capablities at the
originating site (whenever the capability is invoked or presented to the
owner of the RPC object).

The new mechanism effectively equips base-linux with Genode's capablity
model as described in the Chapter 3 of the Genode Foundations book.
That said, the sandboxing of components cannot be assumed at this point
because each component has still direct access to the Linux system-call
interface.

This patch is based on the extensive exploration work conducted by
Stefan Thoeni who strongly motivated the inclusion of this feature into
Genode.

Issue #3581
 2020-04-17 12:40:13 +02:00
Alexander Boettcher
319d2be1af seoul: add vm_stress script version 
Issue #3683
 2020-04-17 12:40:13 +02:00
Alexander Boettcher
29911cf114 seoul: update contrib branch 
- reactivate vga update, range was to small
- remove diagnostic messages
- add support to ignore machine check MSR MCG CAP

Issue #3683
 2020-04-17 12:40:13 +02:00
Alexander Boettcher
8be72b0be1 vbox5: add testing VMs causing load 
Issue #3683
 2020-04-17 12:40:13 +02:00
Emery Hemingway
6359445a8e Add <exit propagate="…"/> to Init config schema 
Fix #3724
 2020-04-17 12:40:13 +02:00
Christian Helmuth
c783764d0b Region-map attach/detach stress depot_autopilot test 
Issue #3715
 2020-04-17 12:40:13 +02:00
Stefan Kalkowski
ff378a8c5b hw: cache maintainance on mapping removal too 
Older ARM processors like ARMv6, or Cortex A8 need to write back changes
of the page-tables to physical ram because the MMU does not use the cache.
This naturally needs to be done not only when adding a mapping,
but on removal too.

Fix #3715
 2020-04-17 12:40:13 +02:00
Norman Feske
03c3040a1d Remove stale documentation 
The topics are either covered by the Genode Founations book for by our
tools, in particular the integration of the prepare_port mechanism with
the run tool.
 2020-04-17 12:40:13 +02:00
Christian Helmuth
cc4e21e7a7 Fix some details in run scripts 
- don't use 'qemu -serial mon:stdio' anymore as it no longer works as
  expected
- use "bash -l" with [terminal] to read user's profile configuration,
  e.g., PATH settings
- added missing boot modules and cap quotas
 2020-04-17 12:40:13 +02:00
Christian Helmuth
bb34aafa45 run: use x-terminal-emulator in [terminal] 
The environment variables TERM and COLORTERM do not contain an
executable names.
 2020-04-17 12:40:13 +02:00
Norman Feske
6dfd268ef1 base/registry.h: remove misleading comment 2020-04-17 12:40:13 +02:00
Christian Helmuth
838c5ba7de ieee754: enable output check for ARM64 
Issue #3723
 2020-04-17 12:40:13 +02:00
Christian Helmuth
4ab990ad5b libc: provide C++ runtime symbols in ABI 
This commit puts all C++ runtime/support symbols of ld.lib.so in a
dedicated section of base/lib/symbols/ld and mirrors the section to
libports/lib/symbols/libc. So, the libc ABI resolves potential C++
runtime dependencies of base-ABI-agnostic components at link time. The
runtime resolution is done by the linker by symbol lookup in ld.lib.so.

Issue #3720
 2020-04-17 12:40:13 +02:00
Alexander Boettcher
13cd25e7b0 vfs/ram: avoid 2x mutex releases in guard 
Issue #3612
 2020-04-17 12:40:12 +02:00
Alexander Boettcher
2b41323fe6 test-* recipes: fix xmllint errors 
config_0.xml.tmp:26: element config: Schemas validity error :
  Element 'config': Character content other than whitespace is
  not allowed because the content type is 'element-only'

Issue #3612
 2020-04-17 12:40:12 +02:00
Norman Feske
7bf47b1982 base-linux: boost max open fds to the hard rlimit 
This patch increases the default limit of the maximum number of open
file descriptors to the hard limit of the system. This is needed for
complex scenarios, which require more FDs than the default of 1024
at core.

Related to issue #3581

Fixes #3721
 2020-04-17 12:40:12 +02:00
Alexander Boettcher
22d71d5a8b os: use Mutex/Blockade 
Issue #3612
 2020-04-17 12:40:12 +02:00
Alexander Boettcher
3956530634 base: use Mutex/Blockade 
Issue #3612
 2020-04-17 12:40:12 +02:00
Alexander Boettcher
e87d60ddf7 core: use Mutex/Blockade 
Issue #3612
 2020-04-17 12:40:12 +02:00
Norman Feske
85a1f91f59 Sanitize Framebuffer::Mode member access 
Fixes #3719
 2020-04-17 12:40:12 +02:00
Alexander Boettcher
dc90740549 nic_router: accept DHCP reply with missing option 
The options are not always set, which throws exception. This leads to not
accepting the selected IP address.

Fixes #3716
 2020-04-17 12:40:12 +02:00
Tomasz Gajewski
d111af922a trace documentation: fix reference to events definitions 2020-04-17 12:40:12 +02:00
Norman Feske
40c21b6d0f window layouter: restore screen tiling 
This commit fixes a regression introduced by "window_layouter: add
dynamic screen handling", rendering the sub-division of screens in
columns and row unusable. The said commit removed a condition needed
for the correct window placement. This patch restores the condition.

Issues #3646
 2020-04-17 12:40:12 +02:00
Norman Feske
4dd110ce5e Remove noux runtime 
The feature set of noux is fully covered by the regular C runtime now.

Fixes #3696
 2020-04-17 12:39:33 +02:00
Norman Feske
d6bdeed38f report_dump: replace noux by plain bash 
This patch also introduces the use of the RTC for creating directory
names and increases the rate of snapshot creation from 10 seconds to
one minute to make sure that directories are named differently when
reading the RTC clock at granularity of minutes.

Issue #3696
 2020-04-17 12:39:33 +02:00
Stefan Thöni
e8fb7ad470 report_rom: add package recipe 2020-04-17 12:39:33 +02:00
Stefan Thöni
5b68286e07 fs_report: add package recipe 2020-04-17 12:39:33 +02:00
Stefan Thöni
bb12d79ae9 vfs: add package recipe 2020-04-17 12:39:33 +02:00
Alexander Boettcher
d657b61f1b x86: support msi-x by platform driver 
Fixes #3699
 2020-04-17 12:39:33 +02:00
Norman Feske
42a77c531a sculpt: remove noux runtime from inspect view 
Issue #3696
 2020-04-17 12:39:32 +02:00
Norman Feske
5347278136 sculpt: perform prepare step without noux 
Issue #3696
 2020-04-17 12:39:32 +02:00
Norman Feske
ca5c25ac04 sculpt: replace log-noux instance by stdin2out 
Issue #3696
 2020-04-17 12:39:32 +02:00
Norman Feske
bb35b997b8 New stdin2out utility as replacement for tail -f 
This little utility allows us to remove the noux instance for the log
view of Sculpt OS.

Issue #3696
 2020-04-17 12:39:32 +02:00
Norman Feske
9c95e4bb4f sculpt: replace noux-based e2fsprogs by fs_utils 
With this patch, Sculpt executes the e2fs file-system utilities
directly instead of hosting them within the noux runtime.

Issue #3696
 2020-04-17 12:39:32 +02:00
Norman Feske
e63195a940 e2fsprogs: disable HAVE_GETMNTINFO 
This patch avoids calls of '__sys_getstatfs' (indirectly via
'getmntinfo'), which does not return any meaningful values in Genode's
libc. Otherwise, the libc's dummy implementation prompts resize2fs to
back out.

Issue #3696
 2020-04-17 12:39:32 +02:00
Norman Feske
0836293d1a libports: src/fs_utils depot archive 
The depot archive contains the utilities of e2fsprogs that are needed
for Sculpt OS as standalone components.
 2020-04-17 12:39:32 +02:00
Norman Feske
330672e030 libports: mke2fs and resize2fs as standalone apps 
This patch makes the e2fsprogs 'mke2fs' and 'resize2fs' available as
standalone components that can be used by Sculpt OS directly without
a Unix emulation environment.
 2020-04-17 12:39:32 +02:00
Josef Söntgen
150d143755 os: use Request_stream API in NVMe driver 
As a result of the API change the memory handling could be simplified.
Since the Block session dataspace is now directly used for DMA, we
actually only have to provide the memory for setting up PRP lists for
large requests (for the moment more than 8 KiB of data).

As we limit the maximum data transfer length to 2 MiB, we get by with
just a page per request. Those memory is allocated beforehand for the
maximum number of I/O requests, which got bumbed to 512 entries. Since
not all NVMe controllers support such large a maximum data transfer
length and this many entries, especially older ones, the values are
capped according to the properties of the controller during
initialization. (The memory demands of the component are around 3 MiB
due to setting up for the common case, even if a particular controller
is only able to make use of less.)

(Although there are controllers whose maximum memory page size is more
than 4K, the driver is hardcoded to solely use 4K pages.)

In addition to those changes, the driver now supports the 'SYNC' and
'TRIM' operations of the Block session by using the NVMe 'FLUSH' and
'WRITE_ZEROS' commands.

Fixes #3702.
 2020-04-17 12:39:32 +02:00
Christian Helmuth
e63c5e6c69 base: rework interplay of signal-proxy and entrypoint 
The former scheme left open a race window between
_process_incoming_signals() and wait_and_dispatch_one_io_signal()
resulting in both threads calling block_for_signal() and blocking
forever with one unprocessed signal.

Fixes #3704
 2020-04-17 12:39:32 +02:00
Christian Helmuth
69080014b0 depot_autopilot: test case for entrypoint 
Test for entrypoint destruction and signalling.

Issue #3704
 2020-04-17 12:39:32 +02:00
Stefan Kalkowski
f051065582 Retire OMAP4 support 
Fix #2850
 2020-04-17 12:39:32 +02:00
Piotr Tworek
585c4b8c69 hw: Add support for Qemu virt arm platforms. 
This patch adds support for booting base-hw kernel on qemu-arm virt
machines. The arm_virt machine has 2GB of RAM, 2 Cortex A15 cores and
uses GICv2 interrupt controller. The arm_64_virt machine also has 2GB of
RAM, but has 4 Cortex A53 cores and uses GICv3. Both machines use PSCI
to boot additional CPU cores.

Fixes #3673
 2020-04-17 12:39:32 +02:00
Piotr Tworek
dad1de1865 hw: Add interface for making basic ARM PSCI calls. 
So far this only exposes two functions allowing the code to turn on
additional CPUs. There is much more functionality defined by PSCI spec
but so far its not needed by Genode.

Fixes #3672
 2020-04-17 12:39:32 +02:00
Pirmin Duss
6f1357c6f8 libc: add symbol used by protobuf 
Issue #3692
 2020-04-17 12:39:32 +02:00
Josef Söntgen
336228f357 os: handle SYNC packet in lx_fs 
Call 'fsync(2)' on files and directories.

Fixes #3711.
 2020-04-17 12:39:32 +02:00
Josef Söntgen
df553e9360 os: remove catch-all from FS VFS update mtime 
Only 'alloc_packet' is supposed to throw an exception, which is already
caught. The 'submit_packet' call does not need further handling.

Issue #3713.
 2020-04-17 12:39:32 +02:00
Josef Söntgen
ddc0caa605 os: remove error messages from FS VFS update mtime 
The update modification timestamp is implemented as one-shot where
the caller is expected to try again if the first attempt failed (see
current libc implementation). So the message is misleading as further
attempts might succeed.

Issue #3713.
 2020-04-17 12:39:32 +02:00
Christian Helmuth
0df358a550 nova: early message on successful boot 
This patch fixes a regression with run/log_core introduced by

  2064ffd64b nova: support multidimensional affinity space

The run tool uses a log message 'run_boot_string' to detect successful
boot as well as to detect unexpected reboots. This message should never
be logged twice and, thus, should not be part of the core_log. The patch
mentioned above moved the former run_boot_string after the core_log
initialization.
 2020-04-17 12:39:31 +02:00
Pirmin Duss
9da428dccf vfs_import: implement write loop for copy_file 
This is required to allow import of large file.

Fixes #3655
 2020-04-17 12:37:26 +02:00
Norman Feske
46fa8197a2 depot: update recipe hashes 2020-03-26 11:39:02 +01:00
Christian Helmuth
52d798ff40 init: support heartbeat in config.xsd 2020-03-26 11:39:02 +01:00
Stefan Kalkowski
256b2fa3e1 hw: cortex a9 diagnostic register fixup 
This is a follow-up fix for commit 202333c881.
It checks for the diagnostic registers being already setup correctly.
Otherwise on platforms with secure firmware, like Pandaboard it will stuck.

Ref #3639
 2020-03-26 11:39:02 +01:00
Norman Feske
d8bcaa4fa4 base-linux: disable ASLR for child processes 
ASLR collides with the libc's fork mechanism on 32-bit. In particular,
the tool_chain_auto.run scenario would sporadically fail while mirroring
the parent's address space.

Fixes #3710
 2020-03-26 11:39:02 +01:00
Emery Hemingway
7a5841b637 ahci: driver improve diagnostic messages for empty ports 
Fix #3709
 2020-03-26 11:39:02 +01:00
Tomasz Gajewski
358fdd54b9 run: multiple power-on-command-param values 
Fixes #3708
 2020-03-26 11:39:02 +01:00
Emery Hemingway
45a74023a7 part_block: support for up to 128 sessions 
Allow sessions for each of the 128 sessions supported by GPT.

Fix #3705
 2020-03-26 11:38:59 +01:00
Alexander Boettcher
6f3f46c7ba top: support affinity space height up to 4 
Issue #3635
 2020-03-26 11:38:59 +01:00
Alexander Boettcher
2064ffd64b nova: support multidimensional affinity space 
Fixes #3635
 2020-03-26 11:38:59 +01:00
Norman Feske
685bd763f9 ldso: place dynamic allocations within linker area 
This is needed to execute the tool-chain scenario on base-nova.
Otherwise, the fork mechanism stumbles upon a region conflict
between ldso allocations and the application heap.

Fixes #3706
 2020-03-26 11:38:59 +01:00
Norman Feske
0f3ac4cb9f base-hw: let upgrade_cap_slab reflect Out_of_caps 
This patch reflects the Out_of_caps exception from core to the client,
in addition to the already covered Out_of_ram exception. It thereby
eliminates a potential abort in core, which I observed with the
tool_chain_auto.run script:

  Kernel: RPC upgrade_cap_slab
  Error: Uncaught exception of type 'Genode::Quota_guard<Genode::Cap_quota>::Limit_exceeded'
  Warning: abort called - thread: entrypoint

In addition to propagating the exception, the patch add the client-side
exception-handling code to the base library.

Fixes #3703
 2020-03-26 11:38:59 +01:00
Norman Feske
208294b0d3 Turn pkg/noux-system into pkg/system_shell 
The new package does no longer require noux.

Issue #3696
 2020-03-26 11:38:59 +01:00
Norman Feske
cf052996d3 Remove pkg/noux-build-x86 
Use-case-specific Unix runtimes should better be realized via Goa now.

Issue #3696
 2020-03-26 11:38:58 +01:00
Norman Feske
4961c84f12 Remove noux from vbox_share.inc 
This snippet is used by the vbox5_win7_64_share.run script.

Issue #3696
 2020-03-26 11:38:58 +01:00
Norman Feske
2f598e5680 Remove noux_uname.run script 
This run script is outdated and not regularly tested. In fact, the
tested mechanism (obtaining values from a '.sysctl/' directory) is
nowhere to be found, neither in noux nor the libc. The test still
returns success as it merely checks for the completion of the sequence,
not the printed results. Hence, it is not worth preserving.

Issue #3696
 2020-03-26 11:38:58 +01:00
Norman Feske
951f484fad Remove noux_gdb.run script 
This interactive run script is not regularly tested nor used.

Issue #3696
 2020-03-26 11:38:58 +01:00
Norman Feske
c086bba71a Remove noux_fs.run 
The test exercised the mounting of an ext2 file system in the noux
runtime. This feature is nowadays routinely stressed by Sculpt OS.

Issue #3696
 2020-03-26 11:38:57 +01:00
Norman Feske
5a02b5fd23 Remove noux_fork.run script 
This run script is superseded by pkg/test-fork.

Issue #3696
 2020-03-26 11:38:57 +01:00
Norman Feske
50a1e540a0 Remove noux_bash.run script 
This run script is superseded by the bash.run script.

Issue #3696
 2020-03-26 11:38:57 +01:00
Norman Feske
b8738dee90 Remove noux_terminal_fs.run script 
This run script tested the terminal VFS plugin as a new feature of noux.
In the meantime, the plugin remains as the only way for the interaction
of command-line-based applications with the terminal session. Hence,
the mechanism is now stressed by many other scenarios.

Issue #3696
 2020-03-26 11:38:57 +01:00
Norman Feske
a7e7eeec38 Remove noux from Vim example scenario 
Issue #3696
 2020-03-26 11:38:57 +01:00
Norman Feske
1d480e3329 Remove noux from tool-chain test 
Issue #3696
 2020-03-26 11:38:57 +01:00
Norman Feske
3b427824a7 ports/make: avoid aliasing with libc symbols 
This patch prefixes a few symbols in the make binary that are offered by
both the libc and the make binary. The clash of symbol names produces
confusing runtime linkage otherwise. This becomes a problem during
fork/evecve.
 2020-03-26 11:38:57 +01:00
Norman Feske
3a1f5bc13a tool/check_abi: prevent use of temporary files 
Issue #3696
 2020-03-26 11:38:57 +01:00
Norman Feske
923c38f7cd libc: allocate errno of main thread statically 
The thread-local errno instance for the main thread must not be
allocated on the application heap because it must survive 'execve'.

Fixes #3701
 2020-03-26 11:38:57 +01:00
Norman Feske
ee6d38a770 libc: prune session labels of forked processes 
This patch reduces the size of session labels for all services other
than LOG, keeping only the last element. This avoids exceeding the
maximum label length in the presence of deep fork hierarchies, e.g., for
running the tool chain.

Fixes #3700
 2020-03-26 11:38:57 +01:00
Christian Helmuth
dd524b56fa libc: propagate socket config to child processes 
Fixes #3698
 2020-03-26 11:38:57 +01:00
Norman Feske
4fe4c9aa63 vfs/pipe: trigger I/O-progress handler on EOF 
When the last writer closes the pipe, we must trigger the I/O progress
handler to deliver the final EOF to the client (READ_OK with out count
of 0).

Fixes #3697
 2020-03-26 11:38:56 +01:00
Norman Feske
e3bc77c386 Remove noux from SSH exec terminal 
This patch replaces the former Noux instance by a sub init that hosts
the combination of VFS + fs_rom + bash. If bash exits, the whole sub
init exits, which implicitly tears down the terminal session used by the
sub init. The latter is expected by the test sequence.

Issue #3696
 2020-03-26 11:38:56 +01:00
Norman Feske
cacbad1bd8 Remove noux from ssh_terminal.run 
Issue #3696
 2020-03-26 11:38:56 +01:00
Norman Feske
cc8552f81e Remove noux_vfs_audit.run 
This run script was merely a development vehicle.

Issue #3696
 2020-03-26 11:38:56 +01:00
Norman Feske
34a60b56f6 Remove noux from fs_query.run 
Issue #3696
 2020-03-26 11:38:56 +01:00
Norman Feske
25f7fdcb40 Remove noux from vfs_import.run 
Issue #3696
 2020-03-26 11:38:56 +01:00
Norman Feske
b6d3063ada Remove noux and cpu_sampler_noux from autopilot 
Issue #3696
 2020-03-26 11:38:56 +01:00
Norman Feske
c2cf0ae8c6 os/vfs.h: fix possible nullptr dereference 
Fixes #3694
 2020-03-26 11:38:56 +01:00
Norman Feske
7f5186d489 vfs/tar: return inode numbers of null records 
The tar vfs plugin returns an inode value of zero and a type directory
for null records, which wrongly triggers the file-system loop detection
of the find utility. This patch returns the node pointer as inode value
instead, which is a unique value.

Fixes #3695
 2020-03-26 11:38:56 +01:00
Norman Feske
e67f389da1 Add fork test to depot autopilot 2020-03-26 11:38:56 +01:00
Norman Feske
4ebc164c11 Add execve test to depot autopilot 2020-03-26 11:38:56 +01:00
Martin Stein
886d0da57c ping_nic_router.run: test dhcp renew 
Ref #3681
 2020-03-26 11:38:56 +01:00
Norman Feske
2912096f6e vfs: distinguish NO_PERM from NO_ENTRY in unlink 
This patch changes the 'Single_file_system' to return NO_PERM only if
the to-be-unlinked file corresponds to the single file. This way, a
<rom> co-mounted with a <ram> file-system does not stand in the way of
unlinking files from the <ram>. The concrete symptom occurred the
following scenario:

  <vfs>
    <dir name="home">
      <ram/>
      <rom name="..."/>
    </dir>
  </vfs>

The following sequence of commands wrongly resulted in "Operation not
permitted":

 $ mkdir -p /home/a/b/c
 $ rm -f /home/a/b/c/d

In this case, rm should not fail (unlink should return ENOENT)

Fixes #3690
 2020-03-26 11:38:55 +01:00
Martin Stein
ba55409c86 nic_router: keep dyn IP when updating domain obj 
When updating the domain object of interfaces that stay with the same domain
during a reconfiguration, until now, the normal "detach raw" function was used.
This caused the old domain object to discard a dynamic IP config as all
interfaces detached. This caused interfaces also to discard network links
established with the old configuration although it wasn't necessary. Thus, now
we use a dedicated "detach" in case that an interface actually stays with its
domain. This new "detach" doesn't decrease the interface counter of the domain,
so, it'll not discard its dynamic IP config. If, however, during a
reconfiguration, there's no interface calling this function (all interfaces
move to another or no domain), the dynamic IP config is still discarded as
expected.

Fixes #3686
 2020-03-26 11:38:55 +01:00
Martin Stein
f3185de7f5 nic_stress: further reduce nr of rounds on sel4 
Otherwise, the limit of caps in Capability_space_sel4 in the NIC router is
reached.

Fixes #3684
 2020-03-26 11:38:55 +01:00
Martin Stein
c1b6000248 nic_router: don't call DHCP client on requests 
Until now, the DHCP client was called also for DHCP requests when an interface
had a domain but yet no IP config. Now, an interface distinguishes between DHCP
requests and replies first and then accordingly calls the DHCP server or the
DHCP client if they're available. This also prevents that the DHCP client has
to handle packet headers other than that of DHCP.

Fixes #3681
 2020-03-26 11:38:55 +01:00
Martin Stein
0a468a07a7 nic_router: fix lifetime of DHCP clients 
Let the DHCP client be a constructible member of Interface that is constructed
only as long as the interface is attached to a domain with a dynamic IP config.
This prevents DHCP client timeouts from a period with dynamic IP config to
trigger after a reconfiguration to a static IP config. Furthermore, handle
DHCP-reply packets at an interface only when the DHCP client its constructed.
Otherwise drop such packets.

Ref #3681
 2020-03-26 11:38:55 +01:00
Stefan Kalkowski
6d48b5484d hw: correct the ARM cache maintainance operations 
This commit fixes the following issues regarding cache maintainance
under ARM:

* read out I-, and D-cache line size at runtime and use the correct one
* remove 'update_data_region' call from unprivileged syscalls
* rename 'update_instr_region' syscall to 'cache_coherent_region' to
  reflect what it doing, namely make I-, and D-cache coherent
* restrict 'cache_coherent_region' syscall to one page at a time
* lookup the region given in a 'cache_coherent_region' syscall in the
  page-table of the PD to prevent machine exceptions in the kernel
* only clean D-cache lines, do not invalidate them when pages where
  added on Cortex-A8 and ARMv6 (MMU sees phys. memory here)
* remove unused code relicts of cache maintainance

In addition it introduces per architecture memory clearance functions
used by core, when preparing new dataspaces. Thereby, it optimizes:

* on ARMv7 using per-word assignments
* on ARMv8 using cacheline zeroing
* on x86_64 using 'rept stosq' assembler instruction

Fix #3685
 2020-03-26 11:38:55 +01:00
Stefan Kalkowski
3a862334fc depot_autopilot: re-enable test-fs_packet on rpi 
Fixes #3249
 2020-03-26 11:38:55 +01:00
Alexander Boettcher
cd1c0aace3 nova: remove specific stack area location 
and use instead the generic one of base

Fixes #3654
 2020-03-26 11:38:55 +01:00
Sebastian Sumpf
6e54cad44c usb_block: switch to Request_stream API 
- remove old 'Driver' interface
- implement Request stream front end
- fix some namespacing

issue #3661
 2020-03-26 11:38:55 +01:00
Josef Söntgen
a71ef16423 vfs: implement 'sync' in Block VFS plugin 
Fixes #3659.
 2020-03-26 11:38:55 +01:00
Sebastian Sumpf
2495a86aff block_tester: add 'sync' command to replay 
issue #3636
 2020-03-26 11:38:55 +01:00
Sebastian Sumpf
5225d12381 ahci: add sync operation support 
- implemement 'flush cache ext' for ATA
- enable sync operation for request stream

issue #3636
 2020-03-26 11:38:54 +01:00
Alexander Boettcher
e096202b1f hw/x86: use PAT for write combining support 
Fixes #3623
 2020-03-26 11:38:54 +01:00
Alexander Boettcher
44ace04b95 hw/x86: add cpuid register macro 
Issue #3623
 2020-03-26 11:38:54 +01:00
Alexander Boettcher
507a34b3d0 hw/x86: implement wrmsr in macro 
Issue #3623
 2020-03-26 11:38:54 +01:00
Christian Helmuth
677d9abd07 rtc: system_rtc pkg for PC and Linux 
This depot package is the runtime for rtc_drv and system_rtc server with
coordinated update of system and hardware RTC via reports. It replaces
drivers_rtc which was never freestanding drivers package and, therefore,
has to be accompanied with a running platform_drv etc.

Fixes #3680
 2020-03-26 11:38:54 +01:00
Christian Helmuth
e66726e931 rtc: use depot archives in test 
Issue #3680
 2020-03-26 11:38:54 +01:00
Christian Helmuth
cda778a94d rtc: recipe for linux_rtc_drv 
Issue #3680
 2020-03-26 11:38:54 +01:00
Sebastian Sumpf
a799a5f978 os: adjust tests to changed output of block client 
issue #3671
 2020-03-26 11:38:54 +01:00
Sebastian Sumpf
f5a6d85bc2 os: expand block_tester.run 
- use two disks with two partitions each
- one disk uses MBR and one GPT
- connect one 'part_block' to each disk
- run 'block_tester' and 'test-block-client' connecting to one partiion
  on each disk
- x86 uses 'ahci_drv' as disk back-end, Linux 'lx_block'
- on autopilot enable for Qemu only

issue #3671
 2020-03-26 11:38:54 +01:00
Christian Prochaska
8535688605 qt5: handle Genode signals as Qt signals in QPA plugin 
Fixes #3688
 2020-03-26 11:38:54 +01:00
Johannes Schlatow
8788f13e11 Qt5 tool chain: always build without zstd 
Fixes #3687
 2020-03-26 11:38:54 +01:00
Johannes Schlatow
aee5b59c51 Backport changeset 82268 from virtualbox 
Fixes #3689
 2020-03-26 11:38:54 +01:00
Christian Helmuth
66a7e749e8 Update URLs in LICENSE 
The official AGPLv3 text from https://www.gnu.org/licenses/agpl-3.0.txt
now uses https:// consistently.
 2020-03-26 11:38:54 +01:00
Norman Feske
e2b9308b11 lx_hybrid_errno.run: add missing parent services 2020-03-26 11:38:54 +01:00
Alexander Boettcher
97bbad5aea nova: avoid dying thread in platform.run 
Avoids timely interference by dying thread.

Fixes #3670
 2020-03-26 11:38:54 +01:00
Christian Helmuth
d452094f43 Only rewind next-free pointer in Bit_allocator 
Formerly, _next was always updated to the last free'd allocation, which
left large gaps on alloc/free bursts. Now, we try keep allocation
density high from the start of the array and ensure that allocations
happen at the lowest available index in the bit array.

Fixes #3679
 2020-03-26 11:38:53 +01:00
Piotr Tworek
139f390317 libc: Expose timespec_get symbol. 
The function is exposed in libc headers provided by Genode, the code for
the function is being compiled and actually works, but the symbol is
missing from the symbols file resulting in linking failures. Add it to
the libc symbols file.

Fixes #3676
Fixes #3677
 2020-03-26 11:38:53 +01:00
Emery Hemingway
9ee1045aad Remove SVN build dependency from libc port 
Fixes #3678
 2020-03-26 11:38:53 +01:00
Norman Feske
25c85fab18 sculpt: version 20.03 
This patch increases the Sculpt version to avoid the risk of binary
compatibility clashes with index files published for the just-released
Sculpt 20.02 version.
 2020-03-26 11:38:53 +01:00
Norman Feske
095a0a6439 News item for Sculpt 20.02 2020-03-10 11:06:10 +01:00
Christian Helmuth
bf36d9eb48 depot: update recipe hashes 2020-03-09 16:02:56 +01:00
Norman Feske
9d24e906a8 text_area: fix interplay of shift with selection 
Without this patch, the text area would mistakenly delete the characters
types after placing the cursor with the mouse as soon as shift is used
as modifier.
 2020-03-09 15:41:29 +01:00
Norman Feske
44e4d1bd6c Update Sculpt documentation for version 20.02 2020-03-09 15:41:29 +01:00
Norman Feske
ab5770c492 sculpt: improve USB hotplug handling 
- Don't reset sculpt partition on USB changes whenever ram fs is
  selected for use.

- Reconfigure runtime immediately when plugging USB devices.
  This prevents the usb-block driver and part_block from
  dangling in the graph. They disappear immediately now.
 2020-03-09 15:36:21 +01:00
Norman Feske
a90eab8b9a menu_view: reduce frame transparency 
This makes the dialog content better readable.
 2020-03-09 15:36:21 +01:00
Norman Feske
e54e4dd532 sculpt: anchor popup position at '+' button 
This patch restores the anchoring of the popup menu to the original
behavior, placing the popup to the right of the '+' button. The
anchoring got lost due to the recent structural changes of the GUI,
which affect the hover XML structure.
 2020-03-09 15:36:21 +01:00
Sebastian Sumpf
c856ba2a49 ahci: report ports correctly 
Only use one report for devices report, not one per device

fixes #3671
 2020-03-09 15:36:21 +01:00
Christian Prochaska
1087e3f59e doc: minor tweaks of release notes 20.02 
Thanks to Jeroen van Gelderen for the feedback.
 2020-03-09 15:36:04 +01:00
Christian Helmuth
cfb49c7316 version: 20.02 2020-02-28 12:19:49 +01:00
Norman Feske
a3fad2e171 News item for version 20.02 2020-02-28 12:19:49 +01:00
Norman Feske
dd5db8484a Release notes for version 20.02 2020-02-28 12:19:49 +01:00
Christian Helmuth
5affd51250 depot: update recipe hashes 2020-02-28 08:59:43 +01:00
Alexander Boettcher
ce27b5ebce tool: support hyphen in depot-user name (depot.inc) 
Fixes #3668
 2020-02-28 08:58:28 +01:00
Norman Feske
fce9cd8c22 sculpt manager: use runtime state report max of 1M 
With complex scenarios, supplementing the <report> node with the
attribute 'requested' or 'provides' can easily result in a report of
more than 64K.
 2020-02-27 14:47:00 +01:00
Christian Prochaska
8faa916d93 gdb_monitor.run: increase RAM quota 
Fixes #3667
 2020-02-27 14:47:00 +01:00
Christian Helmuth
e52802162c libc: reimplement synchronization primitives 
The new implementation relieves the main entrypoint from monitor jobs
for contended lock primitives and is based on custom applicant data
structures, per-lock resp. per-semaphore applicant lists, and a
libc-internal blockade with timeouts based on libc kernel primitives.
 2020-02-27 14:47:00 +01:00
Josef Söntgen
f3ec246b67 base: remove diagnostic message in destructor 
The LOG message in the Signal_handler_thread's destructor is just an
diagnostic leftover.

Fixes #3666.
 2020-02-27 14:46:59 +01:00
Christian Prochaska
434c9ceb5d vbox5-nova: document possible 'Blocking_canceled' exception 
Fixes #3664
 2020-02-27 14:46:59 +01:00
Norman Feske
c3fb81d1a1 window_layouter: improve rules formatting 
This is a follow-up patch for "window_layouter: add dynamic screen
handling".

Issue #3646
 2020-02-27 14:46:59 +01:00
Norman Feske
c340f57207 Xml_node: skip leading whitespace in with_raw_node 
This patch makes the 'with_raw_node' method more useful in situations
where content of an Xml_node is fed into an Xml_generator, i.e., the
rules report/rom mechanism of the window layouter.
 2020-02-27 14:46:59 +01:00
Norman Feske
bbe1bf9c3a fs_query.run: increase timeout 
This is apparently needed for base-foc on Qemu/x86.
 2020-02-27 14:46:59 +01:00
Alexander Boettcher
accda1211b mutex: reinit Applicant state in Mutex::acquire() 
The applicant_to_wake_up() member must be initialized to 0 to keep the
same semantic as before the change by

"base: add mutex as derivate of lock"

Issue #3662
 2020-02-27 14:46:59 +01:00
Norman Feske
daee1f4cb8 timer/nova: prevent potential division by zero 
This case triggered with the leitzentrale.run script on Qemu.
The frequency value must never initialized with zero.

Fixes #3663
 2020-02-27 14:46:59 +01:00
Stefan Kalkowski
87cb10c558 sculpt_manager: accept uplink from "local" network 
Be less rigid when checking for available network to support
setups where the uplink or depot repositories are provided by
another source than "wired" or "wifi" NIC services,
e.g., a virtual machine with pass-through network device.
 2020-02-27 14:46:59 +01:00
Norman Feske
904651ada9 sculpt: version 20.02 2020-02-27 14:46:59 +01:00
Norman Feske
1d3ce93107 sculpt: leitzentrale user-interface redesign 
Issue #3650
 2020-02-27 14:46:59 +01:00
Sebastian Sumpf
103dcdeea8 vmm: ARMv8 virtio improvements 
net:
* increase queue size to 1024 (more stable on Linux)
* use mac address from Nic session instead of random one
* handle data that is larger than rx descriptor correctly (copy less)
* clear descriptor header (12 bytes) on rx

generic:
* always use 'avail_idx' (tx and rx)
* added barriers when reading/writing queues (TMP)

Ref #3620
 2020-02-27 14:46:59 +01:00
Sebastian Sumpf
f77531138a vmm: ARMv8 virtio for network and console 
Simple console and network implementation of the Virtio protocol.

Fix #3620
 2020-02-27 14:46:59 +01:00
Sebastian Sumpf
c8b3b060aa vmm: ARMv8 add 'local_address' to Ram_allocator 
Ref #3620
 2020-02-27 14:46:59 +01:00
Stefan Kalkowski
7780ee6a34 vmm: do not lock inside VMM 
We do not use the VMM with several CPUs until now.
On the other hand there is a dead-lock due to the
generic timer calling the cpu handler function explicitly,
which already holds the lock. For the time-being remove the lock.

Ref #3620
 2020-02-27 14:46:59 +01:00
Stefan Kalkowski
2e2625e952 vmm: check ring-buffer capacity in pl011 model 
Ref #3620
 2020-02-27 14:46:58 +01:00
Stefan Kalkowski
55c3eb7c14 vmm: fix GIC interrupt register bit shifts 
Ref #3620
 2020-02-27 14:46:58 +01:00
Stefan Thöni
a7a9855493 base: lay groundwork for base-linux caps change 
Include the necessary hooks to introduce file descriptor
based capabilities in base linux.

Issue #3581
 2020-02-27 14:46:58 +01:00
Norman Feske
5eaaee0dbe text_area: a simple text viewer / editor 
The new text_area component is able to view and edit files.
Internally, it employs a menu_view for the graphics output.

Only basic notepad-like text-editing functions are supported.

At the current time, it is solely meant as a companion of the Sculpt
manager.

Issue #3650
 2020-02-27 14:46:58 +01:00
Norman Feske
b2bc718c1f fs_query: report 'writeable' attribute for files 2020-02-27 14:46:58 +01:00
Norman Feske
7118ad494c os/vfs.h: add Directory::Entry::rwx accessor 2020-02-27 14:46:58 +01:00
Norman Feske
582e0e718c menu_view: make texture handling more robust 
This patch improves the robustness of menu_view when encounting missing
textures, which can happen during development when using styled buttons
and frames. With the patch, menu_view outputs diagnostic messages,
pinpointing the problem.

The patch also updates the texture handling to use the 'File_content'
utility and the VFS for obtaining PNG images.

Issue #3629
 2020-02-27 14:46:58 +01:00
Norman Feske
1713583a19 menu_view: basic support for styling labels 
This patch allows for the customization of the text color and alpha
value of the label widget by the means of a style-definition file.
The mechanism is exemplified with the new "invisible" label style
that sets the alpha value to zero.

Issue #3629
 2020-02-27 14:46:57 +01:00
Norman Feske
38aef49428 menu_view: invisible button style 2020-02-27 14:46:57 +01:00
Alexander Boettcher
a9caf3fbe4 base: avoid wrong warning by Mutex::acquire 
A _owner is only valid if lock is actually locked.

Issue #3612
 2020-02-27 14:46:57 +01:00
Josef Söntgen
80ff844dc2 depot: update public key for cnuke 2020-02-27 14:46:57 +01:00
Josef Söntgen
c53be5a3fb os: omit size check on payloadless Block packets 
The check is already performed in 'packet_valid' where the conditions
for payloadless packets are already considered.

Fixes #3660.
 2020-02-27 14:46:57 +01:00
Josef Söntgen
6addd6cf1e os: fix Node_rwx::rwx 
Fixes #3658.
 2020-02-27 14:46:57 +01:00
Sebastian Sumpf
3995d2f4a2 part_block: switch to Request_stream and Job APIs 
- use Job API as block connection back end
- use Request_stream API as front end
- use Mmio framework for gpt and mbr parsing
- implement sync correctly

fixes #3652
 2020-02-27 14:46:57 +01:00
Sebastian Sumpf
b95dc611d6 gmp: use architecture specific 'config.h' 
Generated a separate 'config.h' for arm, arm_64, x86_32, x86_64 for the
current version (6.1.2) of GMP. This became necessary because
configurations differ for each architecture.

'config.h' generaton on x86_64 host in'gmp-6.1.2' directory:

for x86_64 (native):
! configure

for x86_32:
! configure --host=x86-pc-linux-gnu --build=x86_64-pc-linux-gnu

for arm:
! configure --host=arm-linux-gnu --build=x86_64-pc-linux-gnu \
!  CC=/usr/local/gcc-linaro-arm/bin/arm-linux-gnueabi-gcc ABI=32

for arm_v8:
! configure --host=aarch64-linux-gnu --build=x86_64-pc-linux-gnu \
!  CC=/usr/local/gcc-linaro/bin/aarch64-linux-gnu-gcc ABI=64

issue #3598
 2020-02-27 14:46:57 +01:00
Christian Helmuth
4cccf74664 depot: update recipe hashes 2020-02-20 12:11:23 +01:00
Stefan Kalkowski
8cc48d5688 hw: be more accurate in synchronizing ASID/Pages 
Fix #3651
 2020-02-20 12:11:23 +01:00
Stefan Kalkowski
b76bd57ed1 window_layouter: add dynamic screen handling 
This commit adds dynamic switching in between different defined
screens via keyboard shortcuts, or by editing the rules file.
In the window_layouter package it defines per default ten screens,
reachable by pressing the screen key along with numbers 0-9.

Fix #3646
 2020-02-20 12:11:23 +01:00
Alexander Boettcher
2afc218767 test/registry: measure time used for iteration 
Issue #3612
 2020-02-20 12:11:22 +01:00
Alexander Boettcher
5bbaa30655 base: use mutex in registry 
Issue #3612
 2020-02-20 12:11:22 +01:00
Alexander Boettcher
5440cd4b50 base: use blockade in thread 
Issue #3612
 2020-02-20 12:11:22 +01:00
Alexander Boettcher
e686ff78e9 base: add blockade as derivate of lock 
Issue #3612
 2020-02-20 12:11:22 +01:00
Alexander Boettcher
2bd77722c7 base: use mutex in heap 
Issue #3612
 2020-02-20 12:11:22 +01:00
Alexander Boettcher
00f69bc70d base: add mutex as derivate of lock 
The mutex class is more restrictive in usage compared to
Genode::Lock.

- At initialiation time it is ever unlocked.
- No thread is permitted to lock twice. Warn about it
  in case it happens.
- Only the lock onwer is permitted to unlock the mutex.
  Warn about it and don't unlock the mutex in case it happens.

Issue #3612
 2020-02-20 12:11:22 +01:00
Stefan Kalkowski
d1609e771a pdf_view: fix typo in pkg runtime requires 
Fix #3648
 2020-02-20 12:11:22 +01:00
Norman Feske
89f813f113 os: add sandbox lib to api/os 2020-02-20 12:11:22 +01:00
Norman Feske
9b0fbf000e sandbox: support for local state-report handling 
Fixes #3647
 2020-02-20 12:11:22 +01:00
Stefan Kalkowski
725d16e18e hw: optimize ipc capability allocation 
In preparation of ipc receive, by now a bunch of capabilities got
allocated to be ready iin case of capability receiption. After that
unuse slots were freed again. This overhead of senseless (de-)allocations
is replaced in this commit by just restock capability slots that got
used by the last receive.

Fix #3640
 2020-02-20 12:11:22 +01:00
Martin Stein
e42a205a51 base-hw: Ada-compliant scheduler method signatures 
Ref #3308
 2020-02-20 12:11:22 +01:00
Martin Stein
0d5f185267 base-hw: no pointers in scheduler interface 
Ref #3308
 2020-02-20 12:11:22 +01:00
Martin Stein
c146a215fb base-hw: cpu scheduler without inheritance 
Ref #3308
 2020-02-20 12:11:21 +01:00
Martin Stein
eef7b5e168 base-hw: double list without inheritance 
Ref #3308
 2020-02-20 12:11:21 +01:00
Martin Stein
a753b6ce46 base-hw: fix run scripts 
Ref #3308
 2020-02-20 12:11:21 +01:00
Martin Stein
793e12f8f3 base-hw: no error return codes in signaling 
Error return codes are used with non-const functions in the signaling modules.
This would have been impractical for the in-place translation of the module to
Ada in the context of the Spunky project. Besides, it is easy to get rid of
them and it makes the execution flow more clear.

Ref #3308
 2020-02-20 12:11:21 +01:00
Martin Stein
751e6430fa base-hw: obsolete Kernel::Thread::_signal_receiver 
Ref #3308
 2020-02-20 12:11:21 +01:00
Martin Stein
9eb20c2be7 base-hw: no pointer arguments in signaling 
Besides reducing the use of pointers in base-hw, this prepares for the in-place
translation of the signaling module to Ada in the context of the Spunky
project.

Ref #3308
 2020-02-20 12:11:21 +01:00
Martin Stein
5e460394d2 base-hw: do not inherit Kernel::Object 
This prevents the use of Kernel::Object as base class for the specific kernel
objects and instead makes it a member of the kernel objects. Besides
simplifying inheritance hierarchies in base-hw, this prepares for the in-place
translation of the signaling module (and others) to Ada in the context of the
Spunky project.

Ref #3308
 2020-02-20 12:11:21 +01:00
Martin Stein
88043e144a base-hw: no virtual functions in signaling 
Besides simplifying the execution flow in the signaling module, this prepares
for the in-place translation of the signaling module to Ada in the context of
the Spunky project.

Ref #3308
 2020-02-20 12:11:21 +01:00
Martin Stein
3cc7774fe4 base-hw: Ada/SPARK-friendly Ipc_node interface 
This prevents the use of C++ features in the public method interface of the
synchronous-IPC module that would be impractical for the in-place
translation of the module into Ada in the context of the Spunky project.

* Get rid of thread accessor.
* Get rid of non-const functions with return values.
* Get rid of pointer return-values.

Ref #3308
 2020-02-20 12:11:21 +01:00
Martin Stein
a04243aaf4 base-hw: remove unused Ipc_node accessor methods 
Ref #3308
 2020-02-20 12:11:21 +01:00
Martin Stein
5a95183c3e base-hw: avoid that Ipc_node inherits 
Besides simplifying inheritance hierarchies in base-hw, this prepares for the
in-place translation of the synchronous-IPC module to Ada in the context of the
Spunky project.

Ref #3308
 2020-02-20 12:11:21 +01:00
Martin Stein
6a5aa18a7b base-hw: no PD code / virt methods in Ipc_node 
This is a simplication of the asyncronous-IPC module of the base-hw kernel.
Besides structuring the code in a cleaner way, it prepares for the in-place
translation of the module into Ada in the context of the Spunky project.

* Get rid of virtual methods in Ipc_node.
* Move all stuff related to protection domains, capabilities, and UTCBs to
  the Thread class. this code might later be moved to a dedicated module, but
  for now it's just fine to have it done by the thread module.

Ref #3308
 2020-02-20 12:11:21 +01:00
Christian Prochaska
4b3c40f35b test-tcp_bulk: increase resources 
Fixes #3645
 2020-02-20 12:11:21 +01:00
Stefan Kalkowski
79fba6c2ac core: add kernel-specific UTCB quota bookkeeping 
Fix #859
 2020-02-20 12:11:21 +01:00
Tomasz Gajewski
4f217b19a9 hw: add CPU wake up code for rpi3 
Moved code waking up processors for Cortex A53 before changing privilege
level because sending events to higher privilege levels is not allowed.

Fixed enable_mmu for Cortex A53 to properly return cpu id.

Fixed starting code for secondary cores to properly initialize stacks.

Added code to wake up secondary cores on rpi3.

Ref #3573
 2020-02-20 12:08:17 +01:00
Stefan Kalkowski
202333c881 hw: cortex a9 diagnostic register initialization 
We cannot count on the correct initialization of the diagnostic register
of the secondary cores. But the boot core gets initialized by the bootchain,
so we can copy over those values.

Fix #3639
 2020-02-20 12:08:17 +01:00
Christian Prochaska
2ce0395fd8 arora.run: fix 'focus' route 
Fixes #3637
 2020-02-20 12:08:17 +01:00
Emery Hemingway
bbdf181828 core: add support for unlabeled LOG sessions 
If the root child requests a LOG service with the label "unlabeled" then
return a LOG session that logs unprefixed messages. This allows a external
test controller to recognize log messages produced by a blessed component.
 2020-02-20 12:08:17 +01:00
Norman Feske
0181c6025a menu_view: add <label min_ex=".."> attribute 
This new attribute can be used to enforce a minimum width in the unit of
the size of the character 'x'. Furthermore, the patch sets the minimum
height of a label to 0 if no 'text' attribute is present. By combining
both features, the label becomes usable as a horizontal spacer.

Issue #3629
 2020-02-20 12:08:17 +01:00
Norman Feske
6eff83c1eb menu_view: invisible frame 
Issue #3629
 2020-02-20 12:08:17 +01:00
Norman Feske
5aae0f2379 menu_view: support multiple floats within a frame 
This patch refines the hover handling such that a float widget never
responds to hovering unless a child is hovered. This way, it becomes
possible to stack multiple float widgets within one frame and still
reach all child widgets.

Issue #3629
 2020-02-20 12:08:17 +01:00
Norman Feske
49ae4a834f menu_view: support for displaying text selections 
Similarly to the way of how a <cursor> can be defined for a <label>
widget, a selection can be expressed as follows:

  <label ...>
    <selection at="2" length="12"/>
  </label>

Issue #3607
Issue #3629
 2020-02-20 12:08:17 +01:00
Norman Feske
5b650434b0 menu_view: unquote label text attribute value 
This patch enables the menu view to display '"' characters supplied via
the 'text' attribute of the <label> widget.

Issue #3607
Issue #3629
 2020-02-20 12:08:16 +01:00
Norman Feske
e612f7cd7d menu_view: add character position to hover report 
When setting the <dialog> attribute 'hover_details' to "yes", the hover
report features the character position of a hovered label.

Issue #3607
Issue #3629
 2020-02-20 12:08:16 +01:00
Norman Feske
7cc4aa2a28 menu_view: text cursor support 
This patch equips the label widget with the ability to display a text
cursor, as illustrated by the following example:

  <label text="...">
    <cursor at="10"/>
  </label>

Issue #3607
Issue #3629
 2020-02-20 12:08:16 +01:00
Norman Feske
60f5d0e34a gems/animated_geometry.h: increase value range 
The value range supported by the 'Animated_rect' utility is constrained
to 2^11 by the used 32-bit integer type. The interpolation is performed
on fixpoint numbers with a fractional precision of 10 bits. The
non-linear interpolation as performed by the 'Lazy_value' utility
involves squaring the values. Hence, the interpolated integral values
must not exceed 2^11 (2*11 bits for the square plus 10 bit for the
fractional part require 32 bits). This range is too small for
high-resolution displays. Hence this patch replaces the 32-bit integer
type by a 64-bit integer type.

Fixes #3627
 2020-02-20 12:08:16 +01:00
Norman Feske
46c5a90ba1 nitpicker_gfx: Text_painter::Font::index_at_xpos 
This method is useful for finding the character under a mouse position.
 2020-02-20 12:08:16 +01:00
Stefan Kalkowski
1273c573b6 wm: destroy view before newly creating it 2020-02-20 12:08:16 +01:00
Norman Feske
3a2895af19 wm: reset pointer position on leave event 
The wm used to remember the last pointer position when observing a leave
event. With such a stale pointer position, the wm would eventually
wrongly create an artifical motion event when a supposedly hovered
window changes its position (during an animated move). This can lead to
the propagation of the stale pointer position to the wm client, which,
in turn, may interpret the outdated position (e.g., the menu view would
report a wrong hover state).

This patch removes the stale state by resetting the '_pointer_pos' when
observing a leave event.

Fixes #3632
 2020-02-20 12:08:16 +01:00
Norman Feske
0bffac6c98 nit_fader: make fading steps configurable 
Fixes #3633
 2020-02-20 12:08:16 +01:00
Norman Feske
c25de5dba3 backdrop: make width and height customizable 2020-02-20 12:08:16 +01:00
Norman Feske
60edfa4d77 fs_query: add directory entries to listing report 
Fixes #3631
 2020-02-20 12:08:16 +01:00
Norman Feske
52e582132f os/vfs.h: handle corner case in join function 
When specifying "/" or "" as rel_path to the 'Directory' constructor,
the constructed directory should refer to the same directory. The
implementation of the join utility did not consider this corner case. It
occurred during the attempt to use fs_query with "/" given as path.

This patch also adds a Directory::Entry::dir accessor that returns true
if the entry is a directory.

Fixes #3630
 2020-02-20 12:08:16 +01:00
Norman Feske
a888041ba4 cached_fs_rom: fix file-handle leak 
Fixes #3634
 2020-02-20 12:08:16 +01:00
Norman Feske
844af06782 base/buffered_output.h: flush at destruction time 
By writing out all dangling characters at destruction time, the
'Buffered_output' utility can be used as a local variable rather
than a long-living object.
 2020-02-20 12:08:16 +01:00
Norman Feske
7da3404bd0 pkg/noux-system: increase nit_fb quota 
With this change, the window can be maximized on a full-HD screen.
 2020-02-20 12:08:16 +01:00
Sid Hussmann
7676f47540 depot_deploy: support heartbeat in deploy config 
Issue #3575
 2020-02-20 12:08:16 +01:00
Josef Söntgen
9d7a58f6a7 dde_bsd: update audio driver to OpenBSD 6.6 
Besides updating to a newer version the update adds the AC97 ICH driver
and addresses shortcomings with the OpenBSD emulation environment:

* Fix 'delay(9)' implementation - it now properly uses 'us' as unit,
  which results on faster initialization of the drivers.

* Fix LOG output that got lost during commit f23579532 and bring over
  the printf implementation from dde_linux for more structured
  printing.

* As said the driver now recognizes the AUICH devices. So far it was
  only tested with the device model in VirtualBox where it produces
  stuttering audio, investigating the cause is still ongoing.

Fixes #3641.
 2020-02-20 12:08:16 +01:00
Christian Prochaska
9bd3d2aa5c qt5: use main entrypoint in QPA plugin 
Fixes #3644
 2020-02-20 12:08:16 +01:00
Christian Prochaska
28e782dda5 qt5: use pthread backend for QThreads 
Fixes #3643
 2020-02-20 12:08:16 +01:00
Christian Prochaska
597098845c libc: support pthread cleanup handlers 
Fixes #3642
 2020-02-20 12:08:16 +01:00
Sebastian Sumpf
8a7deae238 ahci: switch to request stream API + structural changes 
- Remove dated 'Block::Driver' front end and implement
  'Block::Request_stream' front end
- Remove all dynamic memory allocations but DMA memory allocations
- Remove 'Platform_hba' and implement platform specific functions in
  'spec/<platform>/*'
- Ata and Atapi don't inherit from 'Port' any more, but are a member of
  'Port' as a protocol implementation
- Use platform driver for MMIO mappings (x86)
- Exchange stateful initialization of Ata/Atapi with a sequential
  initialization using 'wait_for_any' and 'retry' patterns
- Fix Atapi initialization by setting the byte count limit
- Set FIS receive base only when Cmd::FRE is disabled and Cmd::FR is 0
- Put everything in namespaces ('Ahci', 'Ata', or 'Atapi')
- Ata decides during read/write operations to use native-command queuing
  or normal DMA requests
- Remove port claiming logic (is now done via 'Constructibles')

fixes #3636
 2020-02-20 12:08:16 +01:00
Sebastian Sumpf
73f2c7043c ahci: remove Exynos5 support 
Exynos5 support has ceased on Genode

issue #3636
 2020-02-10 16:00:41 +01:00
Sebastian Sumpf
de24035066 base: add 'wait_for_any' to 'Register_set' 
This is the same as 'wait_for' with the difference that 'wait_for_any'
succeeds when *one* condition is met.

issue #3636
 2020-02-10 16:00:40 +01:00
Norman Feske
57ea1dbdd3 depot: update recipe hashes 2020-02-10 14:29:06 +01:00
Christian Prochaska
9f73f09cec tcp bulk test: add VFS server tests 
Fixes #3611
 2020-02-10 14:21:48 +01:00
Martin Stein
d56b21d329 nic_router: do not change ethernet header as hub 
When forwarding domain-local traffic like a hub, the NIC router shouldn't
modify packets at all.

Fixes #3625
 2020-02-10 14:21:47 +01:00
Christian Helmuth
8d60bc11b5 vbox: periodic timer based on absolute timestamps 
Before timestamps where calculated as durations since current wakeup
which introduces a systematical drift.
 2020-02-10 14:21:47 +01:00
Christian Helmuth
604f4c666b Remove Timed_semaphore from libc incl test package 
Issue #3550
 2020-02-10 14:21:47 +01:00
Christian Helmuth
ff5175ec76 libc: synchronization primitives based on monitor 
The libc monitor facility enables the execution of monitor jobs by the
main thread when the monitor pool was charged. In comparison to the
current suspend/resume_all mechanism the main thread iterates over all
job functions in contrast to waking up all threads to check their
conditions by themselves. Threads are only woken up if the completion
condition was met.

This commit is the result of a collaboration with Christian Prochaska.
Many thanks for your support, Christian.

Fixes #3550
 2020-02-10 14:21:47 +01:00
Christian Helmuth
6aebd5dd95 libc: append serial number to pthread name 
... in pthread_create()

Issue #3550
 2020-02-10 14:21:47 +01:00
Christian Helmuth
3d4bed3374 libc: unify semantic of sleep functions 
sleep(), usleep(), and nanosleep() now return immediately on
zero-timeout. Also, non-zero timeouts sleep at least 1 ms (the current
minimal timeout in libc), which compensates rounding errors.

Issue #3550
 2020-02-10 14:21:47 +01:00
Alexander Boettcher
ee7a77643e depot_deploy: support affinity configuration 
Fixes #3597
 2020-02-10 14:21:47 +01:00
Alexander Boettcher
646d6c368c core: fixup cpu weight on failed thread creation 
Fixes #3613
 2020-02-07 14:14:20 +01:00
Alexander Boettcher
d96e14fe16 sandbox: adjust location size scaling 
by incorporating session location

Issue #1412
 2020-02-07 14:14:20 +01:00
Alexander Boettcher
3a9d450106 trace: avoid page fault on client calling subjects 
Fixes #3608
 2020-02-07 14:14:20 +01:00
Martin Stein
a73ef9fc06 app/ping: fix destination MAC on ARP replies 
The ARP destination MAC was set to the Ethernet source MAC which is not
necessarily the MAC of the other ARP peer. Instead use the ARP source MAC of
the ARP request.
 2020-02-04 16:12:04 +01:00
Stefan Kalkowski
3016b64fac mesa: enable ARM-64 port 
Fix #3622
 2020-02-04 16:10:28 +01:00
Stefan Kalkowski
22498e0b09 os: add backtrace utility for arm_64 
Fix #3621
 2020-02-04 16:10:22 +01:00
Stefan Kalkowski
79dff674fd gems: add connect facility to tcp_terminal 
Instead of listening for new TCP/IP connections only,
the TCP terminal connects to a server when an IP address is
configured in the policy for a terminal client.

Fix #3619
 2020-02-04 16:10:09 +01:00
Stefan Kalkowski
56ef7ca9e7 hw: enable ARMv8 performance counter 
Fix #3618
 2020-02-04 16:05:01 +01:00
Stefan Kalkowski
9db50753f1 libports: add ARMv8 support for libpng 
Fix #3617
 2020-02-04 16:04:54 +01:00
Stefan Kalkowski
e84e1bbf36 dde_linux: optimize refresh routine in imx8_fb_drv 
Fix #3616
 2020-02-04 16:03:08 +01:00
Stefan Kalkowski
fda337a1c0 os: increase throughput of blit for ARM 64-bit 
Fix #3615
 2020-02-04 16:01:35 +01:00
Stefan Kalkowski
f49f91da08 os: increase RAM quota for drivers in fb_bench.run 
Enables fb_bench.run script for new imx8 framebuffer driver.

Ref #3615
 2020-02-04 15:59:33 +01:00
Christian Helmuth
90535a1401 depot: update recipe hashes 2020-02-04 15:51:10 +01:00
Christian Helmuth
43719b5fd1 libc: fix returned timeout-left value 
If the suspend method for the main thread detects that the suspend
condition is false it must return the passed timeout value (not always
0). Otherwise, the caller may incorrectly assume the timeout expired.
 2020-02-04 15:51:10 +01:00
Stefan Kalkowski
2a94f8cdb4 depot: update public key for skalk 2020-02-04 15:51:10 +01:00
Christian Helmuth
1e578f1a50 Log key count in input test 2020-02-04 15:51:10 +01:00
Alexander Boettcher
a036d2373a x86/ps2: disable devices during commands 
Especially the initial reset ended up with an unusable keyboard in rare
cases before. This commit disables keyboard and mouse handling of i8042
during command sequences according to a note from

  https://www.win.tue.nl/~aeb/linux/kbd/scancodes-12.html
 2020-02-04 15:51:10 +01:00
Norman Feske
a2b303e95a Road map for 2020 2020-02-04 15:51:10 +01:00
Alexander Boettcher
7b964fa700 base: add affinity space info to platform info 
Issue #3599
 2020-02-04 15:51:10 +01:00
Alexander Boettcher
72f5f9d133 sculpt_manager: apply affinity-space to runtime 
Issue #3599
 2020-02-04 15:51:10 +01:00
Christian Prochaska
186a6bc080 test-tcp_bulk_lxip: increase timeout value 
Fixes #3609
 2020-02-04 15:51:10 +01:00
Norman Feske
567d9f7910 Xml_node::Unquoted utility for attribute values 
The new utility can be used to revert quoted XML attribute values.
Such quoting is needed whenever an attribute value can contain '"'
characters. E.g., in the menu_view's <label text="..."> widget.

Issue #1757
 2020-02-04 15:51:10 +01:00
Norman Feske
d70cf314d8 Never throw in 'Attached_rom_dataspace::size' 
If the ROM service returned in invalid dataspace, reflect this condition
via a size of zero instead of triggering an exception of type
'Reconstructible<Attached_dataspace>::Deref_unconstructed_object'.

Issue #3606
 2020-02-04 15:51:10 +01:00
Norman Feske
c6445da654 vfs: watch support for <rom> file system 
Fixes #3606
 2020-02-04 15:51:10 +01:00
Norman Feske
96cde52838 vfs: make <rom> fs robust against invalid ROMs 
An invalid ROM dataspace can occur, for example, when requesting a ROM
from the report_rom service before the first report was posted. Such a
<rom> node can still be useful if it's dynamic. E.g., it can be watched.
Hence, the corner case of an invalid ROM dataspace should be gracefully
handled instead of causing an abort.

Issue #3606
 2020-02-04 15:51:09 +01:00
Norman Feske
c67a0d3dd8 os/dynamic_rom_session.h: support Xml_generator 
This patch adds a special variant of a 'Content_producer' called
'Xml_producer', which allows the direct use of an 'Xml_generator'
for generating the ROM-module content.
 2020-02-04 15:51:09 +01:00
Norman Feske
78c0e5f6b6 New sandbox library extracted from init component 
This patch extracts the child-management functionality from the init
component into a new library called "sandbox". The library API is
located at 'os/include/os/sandbox.h'.

The sandbox API allows for the interaction of the component with the
sandboxed children by providing locally implemented services. This
mechanism is illustrated by the new test at os/src/test/sandbox.

Issue #3601
 2020-02-04 15:51:09 +01:00
Norman Feske
f82e7df0ba os/buffered_xml.h: support for generating XML 
This patch extends the 'Buffered_xml' utility with a new constructor
that fills the buffer with the output of an 'Xml_generator'. It thereby
presents an easy way to generate XML to be consumed locally.

The patch also add a deprecation mark to the original 'xml' accessor
because copying 'Xml_node' objects (here as return value) is dangerous.
The new 'with_xml_node' method should instead be used to access the XML
content stored in the buffer.

Fixes #3602
 2020-02-04 15:51:09 +01:00
Norman Feske
640a001ab6 util/xml_node.h: avoid internal use of exceptions 
The XML parser used to rely in C++ exceptions while parsing, which is an
artifact from the initial implementation. This patch reworks the code such
that exceptions are avoided in the common cases.

Fixes #3605
 2020-02-04 15:51:09 +01:00
Alexander Boettcher
c5c5f8754c test-init: add test to check for affinity warning 
Issue #1412
 2020-02-04 15:51:09 +01:00
Alexander Boettcher
5be3bf4f26 init: warn about missing affinity-space 
if affinity is set by children

Issue #1412
 2020-02-04 15:51:09 +01:00
Alexander Boettcher
d132fc0a73 init: scale affinity location 
Issue #1412
 2020-02-04 15:51:09 +01:00
Christian Helmuth
285a33c97d dde_linux: add missing white space in log message 2020-02-04 15:51:09 +01:00
Alexander Boettcher
f09ac23144 base: Timestamp type on arm_64 is 64bit 2020-02-04 15:51:09 +01:00
Alexander Boettcher
734752d6b5 base-nova: show size of framebuffer correctly 2020-02-04 15:51:09 +01:00
Christian Prochaska
4fc6c4ff5c test-tcp_bulk_lwip: increase timeout value 
Fixes #3604
 2020-02-04 15:51:09 +01:00
Christian Prochaska
746d373362 vfs_lxip: classify 'data' file as continuous 
Fixes #3603
 2020-02-04 15:51:09 +01:00
Alexander Boettcher
2256f5fb4b dde_linux: add dma_wmb/rmb barriers for arm_v6 2020-02-04 15:51:09 +01:00
Christian Prochaska
d8e2c95597 vfs server: ensure retry for stalled session 
Fixes #3600
 2020-02-04 15:51:09 +01:00
Alexander Boettcher
6506240642 dde_linux: add dma_wmb/rmb barriers for arm_v7 
and remove re-defined dma_wmb in fec driver
 2020-02-04 15:51:09 +01:00
Sebastian Sumpf
bd284347da libports: fix undefined reference for gmp on arm_v8 
* enable native 'invert_limb' implemenation
* disable native 'udiv_qrnnd' implemenation

fixes #3598
 2020-02-04 15:51:08 +01:00
Sebastian Sumpf
3813f9772a libports: add '__gmpn_invert_limb' for arm_v7 
Implementation was missing.

issue #3598
 2020-02-04 15:51:08 +01:00
Sebastian Sumpf
1902d1a06b hw: do not map boot modules in bootstrap 
This commit safes virtual address space.
 2020-02-04 15:51:08 +01:00
Josef Söntgen
7ecabb25eb ssh_terminal: address ambigouity of return values 
Replace return values with appropiate bool and document two-staged
publickey authentication. This fixes a bug where wrong authentication
attempts are not properly denied.

Issue #3590.
 2020-02-04 15:51:08 +01:00
Alexander Boettcher
5b633a83df core: dissolve thread object on failed creation 
Fixes #3596
 2020-02-04 15:51:08 +01:00
Norman Feske
beb8bf498c base-hw: add explicit array-bounds check 
This patch rules out out-of-bounds array accesses without inspecting the
caller. It is not a bug fix but adds clarity.
 2020-02-04 15:51:08 +01:00
Norman Feske
de764d8490 base-hw/muen: add explicit array-bounds check 2020-02-04 15:51:08 +01:00
Norman Feske
5635c1318c base-hw: explicit bounds check in store_apic_id 
Avoid relying on the caller regarding the CPU index argument to ease the
reasoning about the code.
 2020-02-04 15:51:08 +01:00
Norman Feske
01713c74f9 base-foc: explicit array-index bounds check 
Adding an explicit check clarifies at the first glance that the array
bounds are respected without needing knowledge about MAX_CAPS_PER_MSG
and L4_UTCB_GENERIC_DATA_SIZE (used for dimensioning 'mr').
 2020-02-04 15:51:08 +01:00
Norman Feske
9ec66f0594 allocator_avl: simplify structure 
This patch changes the code of '_alloc_two_blocks_metadata' to not
leak the result of a partial allocation in the out parameters of
the method. This eases the reasoning about the absence of a
use-atfer-free problem (there was none).
 2020-02-04 15:51:08 +01:00
Norman Feske
6947bddd3f base: avoid use of memcpy 2020-02-04 15:51:08 +01:00
Norman Feske
37ec636018 test/smp: add missing destruct call 2020-02-04 15:51:08 +01:00
Norman Feske
9bba6613e7 base/affinity.h: simplify member initialization 
This is just a minor style improvement.
 2020-02-04 15:51:08 +01:00
Christian Prochaska
d4f246517c depot: recipe for drivers_interactive-imx8q_evk 
Fixes #3595
 2020-02-04 15:51:08 +01:00
Stefan Kalkowski
5bfebe7a3f depot: recipe for usb_hid_drv 
Fixes #3594
 2020-02-04 15:47:27 +01:00
Stefan Kalkowski
3df67362b4 depot: recipe for usb_host_drv 
Fixes #3593
 2020-02-04 15:47:27 +01:00
Christian Prochaska
f1042e7fb1 depot: recipe for imx8_fb_drv 
Fixes #3592
 2020-02-04 15:47:27 +01:00
Christian Prochaska
b29112efdf dde_linux: framebuffer driver for i.MX8 EVK 
Fixes #3591
 2020-02-04 15:47:27 +01:00
Stefan Kalkowski
fe899eecc7 base-linux: fix configurable host toolchain 
With the '-x' argument of the 'g++' tool the language gets specified,
therefore it has to be 'c++' furthermore, not CUSTOM_HOST_CXX. Moreover,
when import-lx_hybrid.mk gets evaluated first, global.mk is not included
yet, therefore we cannot eagerly evaluate the CXX_LINK_OPT_NO_PIE variable.

Issue #3466
 2020-02-04 15:47:27 +01:00
Norman Feske
c2a2ec121f libc-net/resolv: silence warning in 3rd-party code 2020-02-04 15:47:27 +01:00
Norman Feske
e1e1fa23b7 zlib: fix compile warning 
This patch includes 'unistd.h' to avoid warnings about the implicit
declaration of the functions ‘read’, 'close', and 'write.
 2020-02-04 15:47:27 +01:00
Norman Feske
aee8d35dc4 libcrypto: suppress warning about unused value 2020-02-04 15:47:27 +01:00
Christian Helmuth
4bbbf5d2e3 dde_linux: reduce compiler warnings 2020-02-04 15:47:27 +01:00
Norman Feske
ba7e832c5d ssh_terminal: fix compile warning 
server.h:51:12: warning: ‘int write_avail_cb(socket_t, int, void*)’
                declared ‘static’ but never defined [-Wunused-function]

This patch fixes the warning by moving the user of 'write_avail_cb' to
the compliation unit that defines it.
 2020-01-03 14:44:31 +01:00
Norman Feske
bbfc092a31 core: avoid use of C array as buffer in Log_root 2020-01-03 14:44:15 +01:00
Norman Feske
de52cf1cdd core: check return value of map.metadata 
Even though the call of map.metadata is known to always return a valid
pointer (because the meta data is assigned in the code just above),
better add an explicit nullptr check.
 2020-01-03 14:44:15 +01:00
Norman Feske
783c05fd6c uart/exynos: be explicit about unused return value 2020-01-03 14:44:15 +01:00
Norman Feske
6ae98e2e6d base: mark Parent::Close_result with [[nodiscard]] 2020-01-03 14:44:15 +01:00
Norman Feske
ffc099eb54 cxx: add nullptr check in calloc 
Even though malloc via the cxx_heap never fails, the implementation of
calloc should better not rely on this assumption.
 2020-01-03 14:44:15 +01:00
Christian Helmuth
9321067b68 Remove unused/redundant longlong.h 
This was used by an ancient libgmp port - longlong.h is now part of the
GMP sources.
 2020-01-03 14:21:41 +01:00
Christian Prochaska
0eaa1f7a08 lx_kit: support enable/disable IRQ and pass IRQ number to handler 
Fixes #3589
 2020-01-02 15:53:46 +01:00
Christian Helmuth
18f90ca1e3 nova: request stack pointer on page fault 
The SP value is part of the Page_fault_info object but was not requested
from the kernel on portal creation.
 2020-01-02 15:30:28 +01:00
Christian Helmuth
9a35743df6 Mark packet-stream descriptor head/tail as volatile 
`volatile` effectively prevents instruction reordering by the compiler
and fixes an issue with -O3 compiled components.

Note, this commit does not address further arguments regarding memory
barriers and volatile voiced in issue #693.

Issue #693
 2019-12-20 14:34:22 +01:00
Christian Helmuth
8d63a3c1f3 Increase timeout of RTC test 
It takes its time on sel4 and Qemu.
 2019-12-20 14:28:18 +01:00
Alexander Boettcher
1ac33caa90 nova: handle rpc_entrypoint destruction better 
Fixes #1921
 2019-12-20 11:31:39 +01:00
Alexander Boettcher
1c361bf545 base: add test to destruct entrypoint 
Issue #1921
 2019-12-20 11:31:27 +01:00
Alexander Boettcher
a41dd48986 nova: avoid invalid capability counting overhead 
- handle it the same way as done for other kernels

Fixes #3579
 2019-12-20 11:29:21 +01:00
Christian Helmuth
b931b67cba depot: update recipe hashes 2019-12-19 17:01:43 +01:00
Christian Helmuth
24435e9ca1 Run TCP bulk tests on all Qemu platforms 
This explicitly includes ARM32/64.
 2019-12-19 17:01:43 +01:00
Alexander Boettcher
e54ff599ca base-hw: trace execution time of core threads 
Fixes #3572
 2019-12-19 17:01:43 +01:00
Sebastian Sumpf
04969b6be0 base-hw: add trace execution time support 
This enables the 'top' program on base-hw for debugging issue #3247 on
rpi.

Fixes #3572
 2019-12-19 17:01:42 +01:00
Christian Helmuth
1ddf1dbc25 Use ram-block in libc vfs tests on Rpi3 2019-12-19 17:01:42 +01:00
Christian Helmuth
8699f5592f wifi: run wpa_main in pthread 2019-12-19 17:01:42 +01:00
Norman Feske
73d089da36 Quota tweaks for base-sel4 2019-12-19 17:01:42 +01:00
Christian Prochaska
c8cd09e72c qt5: don't build qt5_jscore and qt5_webcore for arm_v8a 
Fixes #3585
 2019-12-19 17:01:42 +01:00
Christian Prochaska
be1ef01f10 stdcxx: add symbols needed by dosbox 
Fixes #3588
 2019-12-19 17:01:42 +01:00
Alexander Boettcher
0a1bc1f4b7 nova: add requesting time of killed SCs 
Fixes #3587
 2019-12-19 17:01:42 +01:00
Alexander Boettcher
bcb7f45201 core: avoid dangling ptr in cpu thread component 
if exception is triggered during construction of a thread
 2019-12-19 17:01:42 +01:00
Alexander Boettcher
33db0e0d4d base: add uboot for imx8 as port 
that contains scripts to download and build uboot

Fixes #3584
 2019-12-19 17:01:42 +01:00
Christian Helmuth
504539ad1e vfs/lwip: notify I/O progress on sent packets 
Incoming ACK packets for sent data packets may be the only unblocker for
suspended write/send loops. This patch informs VFS users about I/O of
VFS handle on successfully sent packets.
 2019-12-19 17:01:42 +01:00
Christian Prochaska
a62fce8dc5 sd_card_drv: i.MX8 support 
Fixes #3580
 2019-12-19 17:01:42 +01:00
Christian Prochaska
81a78cf1d0 usb_host_drv: fix 'matches->compatible' string access 
Fixes #3582
 2019-12-19 17:01:42 +01:00
Christian Helmuth
283135c9cd test-libc_getenv: allow unrelated content in log pattern 2019-12-19 17:01:42 +01:00
Sebastian Sumpf
22d4d5c1c1 ldso: update link map during respawn 
move binary to front of link map after respawn. This is required by GDB
in order to load symbol files correctly.

issue #3481
 2019-12-19 17:01:42 +01:00
Sebastian Sumpf
9c372c36c1 libc: thread local errno support 
Store errno in pthread objects, return member upon call to '__error()'.
This became necessary in order to make errno thread-safe.

Note, any call to libc code from a non-pthread (beside the first
entrypoint) is not supported.

issue #3568
 2019-12-19 17:01:42 +01:00
Sebastian Sumpf
9767c4db0e vfs_lwip/lwip: Lwip::lock for concurrent access 
This is required in case the send and receive of IP packets is
performed by multiple thread because lwip is not thread safe.

issue #3568
 2019-12-19 17:01:42 +01:00
Alexander Boettcher
9812799b24 seoul: reduce CPU load in graphic mode 
- Stop refreshing if all CPUs go to sleep, also in text mode.
- Any input resets update rate to 100Hz in graphic mode, which gets
  decreased step by step down to 25Hz if no input is available (looking video
  without input by user).

Fixes #3576
 2019-12-19 17:01:42 +01:00
Alexander Boettcher
d385749ead seoul: make vmm memory configurable 
- document Genode specific configuration option
 2019-12-19 17:01:42 +01:00
Christian Helmuth
23ed5d3936 base-linux: configurable host toolchain 
This commit uses CUSTOM_HOST_CC/CUSTOM_HOST_CXX instead of hardcoded
commands and introduces HOST_DEV_PREFIX.

Original patch by Roman Iten and Pirmin Duss.

Issue #3466
 2019-12-19 17:01:42 +01:00
Christian Helmuth
cebc963396 Cleanup linux_rtc_drv and lx_fs target.mk 
Original patch by Roman Iten and Pirmin Duss.

Issue #3466
 2019-12-19 17:01:42 +01:00
Norman Feske
0c8ec41c21 vfs/pipe: let num_dirent return 0 
This prevents the original value ~0UL from messing up the output
of 'ls -l /dev'.

Issue #3578
 2019-12-19 17:01:42 +01:00
Norman Feske
9f7b8c1a17 vfs/log: allow output redirection via '> /dev/log' 
Issue #3578
 2019-12-19 17:01:42 +01:00
Norman Feske
cd92b32622 libc: close all open FDs on exit 
This is important to issue sync requests for written-to files.

As the closing must be performed by an atexit handler, it happens at a
time _after_ libc plugins are destructed. Consequently an FD allocated
by such a plugin results in a close error, which in turn, does not
destruct the FD. We ultimatedly end up in an infinte loop of
re-attempting the close. For this reason, the patch changes 'close' to
be robust against this special case.

This is generally not a problem because libc plugins are phased out.
However, at present, the libc_noux plugin is still important. With the
changed 'close' in place, there occurred an error message "Error: close:
close not implemented" at the exit of each noux program. This patch
removes the error printing from the libc plugin mechansim to avoid this
noise. The error messages are not important anyway because the
deprecation of the libc plugin interface.

Issue #3578
 2019-12-19 17:00:48 +01:00
Norman Feske
5853a68904 libc: silence calls of getpeername w/o socket fs 
The getpeername function is provided only by the socket fs.
In the case where the socket fs is not configured, return an appropriate
errno instead probing for a libc plugin (there is none).

Issue #3578
 2019-12-19 17:00:48 +01:00
Norman Feske
ae64830bd5 libc: silence get(e)gid, get(e)uid, getppid 
These dummies spam the log when running bash + make, like in the
genodians.org scenario.

Issue #3578
 2019-12-19 17:00:48 +01:00
Norman Feske
e8878eee8a ldso: increase initial heap block 
This is needed to enable the fork of 'make' in the genodians.org
scenario on NOVA.

Issue #3578
 2019-12-19 17:00:48 +01:00
Norman Feske
3897ddea03 cxx: don't rely on global ctors 
This patch removes the global variable 'blocker', which was expected to
be constructed via the global ctors. This mechanism, however, is not
used for the base library, which resulted in the use of an unconstructed
object. Specifically, the spinlock of the 'Lock' of the 'Registry'
defaulted to the LOCKED state (value 0), which eventually would lead to
a deadlock in the contention case of the cxa guard.

I could observe this deadlock once on during the component startup on
base-linux during the construction of the 'startup_lock'.

This patch fixes the problem by explicitly initializing the registry
of blockers via an init function.

Issue #2299
Issue #3578
 2019-12-19 17:00:48 +01:00
Norman Feske
6858270517 base: disarm atexit handling for signal thread 
The signal thread is not supposed to be destructed in any scenario other
than the noux fork mechanism (where no signals occur).

Issue #3578
 2019-12-19 17:00:48 +01:00
Norman Feske
4e57b6eceb core: fix destruction of dependent CPU sessions 
Issue #3578
 2019-12-19 17:00:48 +01:00
Norman Feske
298f317f44 core: remove quota-related diagnostic messages 
Issue #3578
 2019-12-19 17:00:48 +01:00
Norman Feske
5820ad8309 fiasco: silence ipc_send error messages 
These diagnostic messages spam the log but can occur during the regular
component-destruction procedure.

Issue #3578
 2019-12-19 17:00:48 +01:00
Norman Feske
b7fbe65ff2 libc: fork/execve improvements 
- Close FDs marked with the close-on-execve flag
  (needed for 'make', which sets the flag for the pipe-in
  FD of forked children)
- Update binary name on execve to use as ROM for subsequent fork
- Enable vfork as an alias for fork (needed by make)
- Purge line buffers for output streams during execve because they
  may be allocated at the allocation heap, which does not survive
  the execve call.
- Consider short-lived processes that may exit while the parent still
  blocks in the fork call.

With these changes, the website generator of genodians.org works without
the need for the Noux runtime.

Issue #3578
 2019-12-19 17:00:47 +01:00
Norman Feske
d1cf216384 libc: propagate pipe config to child processes 
Issue #3578
 2019-12-19 17:00:47 +01:00
Alexander Boettcher
3011dc5876 sculpt: add vbox5 generic package as option 
Fixes #3574
 2019-12-19 17:00:47 +01:00
Alexander Boettcher
6e99f00f5c usb_hid_raw: enable imx8q_evk for nightly testing 
Issue #3571
 2019-12-19 17:00:47 +01:00
Alexander Boettcher
beb1e084a6 base: add imx8 evk usb driver to platform config 
Issue #3571
 2019-12-19 17:00:47 +01:00
Alexander Boettcher
e34b443c29 usb: add imx8q_evk support to usb_host 
Issue #3571
 2019-12-19 17:00:47 +01:00
Alexander Boettcher
6b17bb647e dde_linux: add barriers to readl/writel 
Issue #3571
 2019-12-19 17:00:47 +01:00
Christian Helmuth
4299b85cdb base-linux: support invalid dataspace for RM client 
This restores behavior introduced in

  commit cb232891bf
  Author: Norman Feske <norman.feske@genode-labs.com>
  Date:   Tue May 10 11:55:25 2016 +0200

    Fix noux.run on base-linux

but removed by issue #2829.

Related to issue #1938
 2019-12-19 17:00:47 +01:00
Sebastian Sumpf
6dae147785 libc: limit fd id allocations to FD_SETSIZE 
fd > FD_SETSIZE cannot use 'select' or 'poll' within our libc.
Therefore, we added a bit allocator in order to allocate fd < FD_SETSIZE
(1024).

fixes #3568
 2019-12-19 17:00:47 +01:00
Sebastian Sumpf
e4255e4c8b vfs/lxip: wait for i/o progress on connect 
Since connect is asynchronous, add a waiter during connect

issue #3568
 2019-12-19 17:00:47 +01:00
Christian Helmuth
161274f785 pistachio: check for 'ed' tool on port prepare 
Some debian systems seems to come without 'ed' in the base installation.
The tool is needed when building the pistachio kernel.
 2019-12-19 17:00:47 +01:00
Roman Iten
6145cdcf37 run: close connection to spawned processes on kill 
From the man page of expect:

> Both expect and interact will detect when the current process exits
> and implicitly do a close.  But if you kill the process by, say, "exec
> kill  $pid",  you  will need to explicitly call close.

Fixes #3569
 2019-12-19 17:00:47 +01:00
Tomasz Gajewski
b57a4c98cf tool/run: run external command plugin for power_on subsystem 
Fixes #3385
 2019-12-19 16:59:03 +01:00
Emery Hemingway
a3e43aca87 Clang: undeclared identifier __UINT64_C 
Issue #3564
 2019-12-19 16:59:03 +01:00
Norman Feske
6b6915e304 ttf_font: fix artifacts with large font sizes 
This patch is a follow-up fix for "ttf_font: prevent out-of-bounds
access", which produced sporadic single-pixel artifacts with large font
sizes, e.g., with the monospaced font set to 24px in Sculpt.

The artifacts occurred only for some horizontal sub-pixel positions, in
combination with the font cache, and after the rendering of '>'
characters. They were ultimately caused by the missed clearning of the
first pixel of a glyph where x0 is 1 (e.g., the character 'd'). In this
case, a pixel from the previously evicted cache entry (the '>') shined
through. The patch fixes the problem by clearing the glyph starting from
the first, not the x0's, pixel.

Fixes #3567
 2019-12-19 16:59:03 +01:00
Alexander Boettcher
3f83ac5580 tool: add imx8_evk board variable hint 2019-12-19 16:59:03 +01:00
Emery Hemingway
7f57de1b74 VFS/terminal: fix clang warning 
Variable-sized object may not be initialized.

Issue #3564
 2019-12-19 16:59:03 +01:00
Emery Hemingway
648382db74 Align after template expansion 
Clang Cannot align template structs. Also, cannot cast void* to addr_t
in constexpr function.

Issue #3564
 2019-12-19 16:59:03 +01:00
Emery Hemingway
2c510bb7f9 Remove unused lamba capture to fix clang warning 
Issue #3564
 2019-12-19 16:59:03 +01:00
Emery Hemingway
23710dff5e Remove dead code 
Issue #3564
 2019-12-19 16:59:03 +01:00
Emery Hemingway
ff0436357b nic_stress: keep signal-context cap by value 
Fixes clang warning about binding reference to stack allocated
parameter.

Issue #3564
 2019-12-19 16:59:03 +01:00
Emery Hemingway
091e5157aa vfs server: fix File_system namespace warnings 
Issue #3564
 2019-12-19 16:59:03 +01:00
Alexander Boettcher
c1e181a407 vbox: handle invalid dataspace exception 
Fixes #3566
 2019-12-19 16:59:03 +01:00
Stefan Kalkowski
8f71c90ca8 dde_rump: backport aarch64 (Fix #3565) 2019-12-19 16:59:03 +01:00
Norman Feske
f85ec313de os/vfs.h: handle zero-sized files 
The 'File_content' class failed to consider zero-sized files.

Fixes #3557
 2019-12-19 16:59:03 +01:00
Norman Feske
2aa6471608 test/depot_query: add test for zero-sized archives 
This test covers the situation where depot_query evaluates depot content
that is incompletely extracted. In particular, if the 'archives' file
already exists but has a size of zero, depot_query would produce a
page fault. This situation can be manually provoked by deliberately
creating a zero-sized archives file for any otherwise correct pkg.

This patch also fixes the success indicator of the test. It wrongly
matched an early line of the log output.

Issue #3557
 2019-12-19 16:59:02 +01:00
Stefan Kalkowski
9814fc5447 libports: enable arm_64 libssl, libcrypto variant 
Fix #3563
 2019-12-19 16:59:02 +01:00
Martin Stein
3655ea77a3 depot_autopilot: rework log matching 
* The log history of a test is stored as a whole for the lifetime of the test.
* Matching of the log history against log patterns is done correctly now
  (previously, a pattern like "AAB" on an input like "AAAB" wouldn't have
  triggered).
* Use memcmp, memcpy, memmove instead of the former character-wise operations.
* Sanitizing of log input and log patterns now works more generic through the
  new Filter class for all replacements/removals.
* Sanitizing is done as soon as a string is available and remains for the
  lifetime of the test.
* Sanitizing doesn't interfer with the matching algorithm.
* Decomposing into small clearly named functions.

Ref #3555
 2019-12-19 16:59:02 +01:00
Johannes Kliemann
89d35bc41e tool: boot Genode on bare metal Linux 
Fixes #2829
 2019-12-19 16:59:02 +01:00
Johannes Kliemann
93639532f0 base-linux: core session support (IO_PORT, IO_MEM, IRQ) 2019-12-19 16:59:02 +01:00
Christian Helmuth
8cf7aaad65 version: 19.11 2019-11-28 11:49:10 +01:00
Norman Feske
25a8ef3b7c News item for version 19.11 2019-11-28 11:44:29 +01:00
Norman Feske
b18a56c2c4 Release notes for version 19.11 2019-11-28 11:44:29 +01:00
Christian Helmuth
9d42e3f69b release notes: fix example in 19.08 input section 2019-11-28 11:44:29 +01:00
Christian Helmuth
11ef8e1ff2 depot: update recipe hashes 2019-11-28 09:06:39 +01:00
Christian Helmuth
1deab4a67f Disable noux_tool_chain_auto on foc/imx7d_sabre 
The bootstrapper always fails with

  fatal: could not find free RAM region for modules

in this scenario.
 2019-11-28 09:06:27 +01:00
Christian Helmuth
f23c70e068 sculpt_manager: use Codepoint::valid() 2019-11-28 09:06:26 +01:00
Alexander Boettcher
f7c818d303 vbox5: add recipe of kernel independent version 2019-11-25 16:02:40 +01:00
Stefan Kalkowski
7996cf06ab os: remove jitterentropy from fs_packet test 
Fixes #3562
 2019-11-25 15:44:57 +01:00
Stefan Kalkowski
4800bcf5a0 hw: correct the i.MX6 Sabrelite timer settings 
Fix #3561
 2019-11-25 15:43:59 +01:00
Christian Prochaska
4c74f4792c ahci_drv: set 'Cmd::St' to 1 before writing 'Sact' register 
Issue #3058
 2019-11-25 14:15:57 +01:00
Christian Prochaska
57d080d4f8 hw: use correct type on IRQ kernel object destruction 
Fixes #3560
 2019-11-25 14:15:39 +01:00
Alexander Boettcher
2778debc29 ram_fs: close open nodes on session close 
Fixes #3559
 2019-11-22 15:59:13 +01:00
Martin Stein
7182c10c90 test/nic_stress: test bad session args 
Fixes #3525
 2019-11-22 14:16:24 +01:00
Christian Prochaska
7309bcf4b5 gdb_monitor: add platform checks in run scripts 
Fixes #3558
 2019-11-22 14:12:06 +01:00
Sebastian Sumpf
309bc2083e vbox5: raise rrtimer priority 
Make sure timers run at the same priority as component threads, otherwise
no timer progress can be made. See 'rtTimeNanoTSInternalRef' (timesupref.h)
and 'rtTimerLRThread' (timerlr-generic.cpp).
 2019-11-22 13:33:15 +01:00
Norman Feske
6e098a9d17 ram_fs: apply session policy to write / mtime ops 
This patch extends the distinction between read-only and writeable
sessions to packet-stream operations for WRITE and WRITE_TIMESTAMP.
 2019-11-21 16:21:19 +01:00
Norman Feske
573b6d3345 noux: fix condition for mtime update 
This is a follow-up patch for "noux: don't update mtimes for read-only
files". It eliminates warning messages during Sculpt's prepare step when
'cp' tries to update the mtimes of the source files.
 2019-11-21 16:21:19 +01:00
Norman Feske
077fa355ce pkg/wm: increase RAM quota 
This patch avoids a warning during the wm startup in Sculpt.
 2019-11-21 15:13:20 +01:00
Norman Feske
e76ce05844 sculpt manager: ignore empty deploy configs 
When copying config/managed/deploy to config/deploy, the latter may
temporarily result in an empty configuration. Such an intermediate
state should be ignored to keep the currently running scenario in tact
instead of restarting it.
 2019-11-21 15:07:51 +01:00
Norman Feske
4622ddb46f nitpicker: add version attribute in clicked report 
This patch addresses a corner case where the nitpicker focus is not
solely defined by mouse clicks or (exclusively) by a window manager, but
by a policy component that takes mouse clicks and other policy (e.g.,
a lock screen) into account. It ensures that each click that follows a
focus change (however initiated) results in a new "clicked" report even
when the report looks the same. To allow the policy component to
uniquely distiguish subsequent reports, the report features a new
'version' attribute.

Fixes #3493
 2019-11-21 14:31:52 +01:00
Stefan Kalkowski
972e1893c9 os: introduce vmm component for ARMv8 
Fix #3553
 2019-11-21 14:29:36 +01:00
Stefan Kalkowski
af29dcf557 hw: introduce virtualization support for ARMv8 
Ref #3553
 2019-11-21 14:29:36 +01:00
Stefan Kalkowski
f82714f341 vm_session: return vcpu id when creating vcpu 
Ref #3553
 2019-11-21 14:29:36 +01:00
Stefan Kalkowski
02d68fdb97 hw: move arm virtualization to generic place 
Ref #3553
 2019-11-21 14:29:36 +01:00
Stefan Kalkowski
065b9fdb46 base-hw: extend syscalls to five arguments 
Ref #3553
 2019-11-21 14:29:36 +01:00
Stefan Kalkowski
18dbd75860 base: explictly target entrypoint in timer session 
Fix #3548
 2019-11-21 14:29:36 +01:00
Norman Feske
3ac970ac1d Streamline Timestamp::INVALID values 
This is a follow-up patch for issue #1784 that solves two
inconsistencies.

- The Vfs::Timestamp::INVALID matches File_system::Timestamp::INVALID
- The Noux libc plugin tests for Timestamp::INVALID instead of a
  positive value.

The patch fixes the mtime info as shown in directory listings in
Sculpt's inspect window.
 2019-11-21 14:26:27 +01:00
Norman Feske
3aaed7188f ram_fs: tie writeable bit to session policy 
Related to issue #3507
 2019-11-21 14:26:01 +01:00
Norman Feske
ee64e29e77 vfs server: tie writeable bit to session policy 
This patch let the VFS server reflect the session policy via the
writeable bit in directory entries instead of merely forwarding the bit
from the respective VFS plugin. This way, all files originating from a
read-only file-system session automatically appear in directory listings
as read-only files.

Related to issue #3507
 2019-11-21 14:25:43 +01:00
Norman Feske
cfba429c15 vfs/rump: offer all files as read-writeable 
The distinction between read-only and read-writeable is the
responsibility of the front end, i.e., the vfs server.
 2019-11-21 14:25:05 +01:00
Josef Söntgen
25aa25c6a0 wifi_drv: enable loading of 9000 series FW 
Tested with a 'Intel(R) Wireless-AC 9462' device.

Fixes #3556.
 2019-11-20 12:56:10 +01:00
Josef Söntgen
2afc02051c sculpt_manager: increase Rom_name size 
The current value of 32 is too short for accessing the
iwlwifi 9000 ucode ROM.

Issue #3556.
 2019-11-20 12:56:03 +01:00
Christian Helmuth
ce1b813105 depot: update recipe hashes (vbox, sculpt) 2019-11-19 16:08:19 +01:00
Christian Helmuth
7ed1d7f11d depot: update recipe hashes 2019-11-19 14:54:14 +01:00
Norman Feske
6ccd65bd8e sculpt_manager: invoke mkfs.ext2 with -F 
The block-vfs plugin is no longer a special file. So we have to
explicitly discharge the sanity check of the mkfs tool.

Issue #3507
 2019-11-19 14:54:14 +01:00
Norman Feske
18b621c2fe sculpt: version 19.11 2019-11-19 14:54:14 +01:00
Alexander Boettcher
cb61a28362 vbox5: use with_libc when call-in into vbox code 
required due to pthreads becoming/are an integral part of the libc now.
Commit avoids libc error:

Error: libc suspend() called from non-user context (0xaf5c784) - aborting

Issue #3550
 2019-11-19 14:54:14 +01:00
Norman Feske
4d7d208940 test-libc_vfs_fs_chained: fix XML validity error 
This problem was spotted by the test.run script.
 2019-11-19 14:54:14 +01:00
Sebastian Sumpf
f7509a5b78 libports: libc: extract file name from path in dlopen 
'dlopen' causes the ldso to open ROM connections, right now we only
support single file names for these ROM not paths. Therefore, we extract
the file name from path within libc's 'dlopen'

fixes #3551
 2019-11-19 14:54:14 +01:00
Christian Helmuth
a54c04d247 libc: return EPIPE on send to shut down sockets 
This intermediate fix still lacks the generation of SIGPIPE and return
of EINTR if the MSG_NOSIGNAL flag is not set in the call to send().
 2019-11-19 14:54:14 +01:00
Christian Helmuth
e70c04ef86 lwip: shut down TCP sockets return WRITE_ERR_IO on write 2019-11-19 14:54:13 +01:00
Christian Helmuth
6410e88698 netperf: handle EPIPE / silence warnings 
This is an intermediate fix to handle sends to sockets shut down by the
peer already. A final fix in the libc would add support for the
MSG_NOSIGNAL send flag, generate SIGPIPE, and return the expected EINTR.
Then this patch is no longer required.

This commit also drops the obsolete patch for missing
sysconf(_SC_NPROCESSORS_ONLN), which is now part of libc.
 2019-11-19 14:54:13 +01:00
Christian Helmuth
a3cb9d9897 lwip: prevent pointer-compare compilation warnings 2019-11-19 14:54:13 +01:00
Alexander Boettcher
b2a7ac2996 depot_deploy: support cpu quota configuration 
the same as the ram and cap quota
 2019-11-19 14:54:13 +01:00
Norman Feske
5f350adb57 vfs/lxip: don't misinterpret EINPROGRESS as error 2019-11-19 14:54:13 +01:00
Norman Feske
1485cd9d24 noux: don't update mtimes for read-only files 
The VFS server as reworked in issue #3547 responds to such an attempt
with a failure as acknowledgement, triggering the client-side error
"Error: packet operation=2 failed" at noux.
 2019-11-19 14:54:13 +01:00
Norman Feske
65d72fb07a drivers/gpu/intel: upgrade platform session on demand 2019-11-19 14:54:13 +01:00
Norman Feske
4871c7bba0 drivers/gpu/intel: reflect insuff. quota to client 
This patch prevents the driver from translating Insufficient_ram_quota
and Insufficient_cap_quota exceptions to Service_denied exceptions at
session-creation time. This enables the client to respond to such a
condition by retrying the session creation with increased session quota.
 2019-11-19 14:54:13 +01:00
Norman Feske
97e2968986 pkg/test-tcp_bulk_lwip: remove lwip.lib.so 
The lwip library is not shared object.

This change is needed to execute the test with the test.run script.
 2019-11-19 14:54:13 +01:00
Norman Feske
eb9a9bf23d bash.run: support using pipes 2019-11-19 14:54:13 +01:00
Emery Hemingway
c51b4b5742 New VFS plugin for emulating POSIX pipes 
Add a new plugin for creating pipes between pairs of VFS handles. It is
intended to replace the libc_pipe plugin, one of the last remaining libc
plugins.

In contrast to the libc_pipe plugin, this plugin defers cross-handle
notification until I/O signal handling rather than block and unblock
readers using a semaphore. This is a performance regression in the case
of multiple threads blocking on a pipe, but shall be an intermediate
mechanism pending renovations within the libc VFS and threading layers.
As a side effect, threads blocked on a pipe might not be resumed until
the main thread suspends and dispatches I/O signals.

The "test-libc_pipe" test has been adjusted to use the VFS pipe plugin
and tests both local pipes and pipes hosted remotely in the VFS server.

Merge adaptations (such as EOF handling, adjustment to VFS/libc
interface changes) by Norman Feske.

Fix #2303
 2019-11-19 14:54:13 +01:00
Norman Feske
9b7915facb vfs server: redesigned internal control flow 
This patch reworks the VFS server to become easier to understand.

- The packet-stream handling is hidden from the node types.

- Introduced the notion of a "job" as an abstraction from the raw packet
  stream.

- The processing of requests is split into three phases:
  (1) accepting new jobs,
  (2) the execution of jobs, and
  (3) the delivery of acknowledgements.

- There is no longer a global fifo of pending nodes. Instead, each
  session has a fifo of "active" nodes, which are nodes that wait
  for the completion of jobs.

- The Io_node type no longer uses virtual functions as hooks for
  the derrived classes. Instead, it provides a number of utility
  functions.

- The control flow of the 'Io_progress_handler' is now the same as
  for the session-local packet-stream handler.

- Removed dependency from 'file_system/node.h', which is a relic
  from the ram_fs.

While reworking the code, the following semantic changes were made
additionally.

- The handling of SEEK_TAIL no longer overwrites the file content at the
  end of the file.

- Removed diagnostic message at session-creation time.

- Write loop for continuous files

Issue #3547
Issue #2303
 2019-11-19 14:54:13 +01:00
Norman Feske
f0de187bbb libc: increase default quota of forked processes 
This patch reduces the debug noise for the prominent case of executing
bash with coreutils. Without it, the forked process will always ask for
more RAM immediately when starting up.
 2019-11-19 14:54:13 +01:00
Norman Feske
539110c0b1 base: make 'Fifo::Element::enqueued' const 
Issue #3547
 2019-11-19 14:54:13 +01:00
Norman Feske
d7b1a89087 file-system session: use same dirent size as VFS 
This change allows for the conversion of directory entries in place
(i.e., in the VFS server) and anticipates the potential future merge of
both types into one.

Issue #3547
 2019-11-19 14:54:13 +01:00
Christian Helmuth
c50252fb35 libc: suspend/resume in pthread mutex lock/unlock 
Issue #3550
 2019-11-19 14:54:13 +01:00
Christian Helmuth
54002e6e6b test-pthread: refactor and cleanup test cases 
Issue #3550
 2019-11-19 14:54:13 +01:00
Sebastian Sumpf
16994d637b libports: update gcov port 
removes bogus include from generated file libgcc_tm.h

issue #3537
 2019-11-19 14:54:12 +01:00
Martin Stein
b541a0d448 smbios.h: move to os/include/smbios 
Fixes #3489
 2019-11-19 14:54:12 +01:00
Norman Feske
d4e0d2f578 libc: defer clone ack after FD initialization 
Issue #3478
Issue #2303
 2019-11-19 14:54:12 +01:00
Norman Feske
fab2fc874f libc: write loop for continuous files 
This patch improves the libc's write operation to iterate on partial
writes to continuous files until the original write count is reached.
The split of large write operations into small partial writes as
dictated by the VFS infrastructure (e.g., constained by I/O buffer
sizes) becomes invisible to the libc-using application.

Issue #3507
Issue #2303
 2019-11-19 14:54:04 +01:00
Sebastian Sumpf
54643d6878 base: add entry to .dynanmic section in genode_dyn.ld 
Make sure that at least entry is present in the .dynamic section, so the
PT_DYNAMIC segment points to something valid in case there are no
dynamic symbols.

issue #3537
 2019-11-19 14:45:40 +01:00
Sebastian Sumpf
2653fad0c4 ports: noux_tool_chain_auto for ARMv8 
issue #3537
 2019-11-19 14:45:40 +01:00
Sebastian Sumpf
b752d22a77 libports: checkout ARMv8 version of gcov 
issue #3537
 2019-11-19 14:45:39 +01:00
Sebastian Sumpf
ad12b42d1c gems: depot autopilot enable gcov for ARMv8 
issue #3537
 2019-11-19 14:45:39 +01:00
Sebastian Sumpf
c845a2d943 libprots: libgcov ARMv8 support 
issue #3537
 2019-11-19 14:45:39 +01:00
Sebastian Sumpf
b7e06a0b5b recipe: gmp for ARMv8 
issue #3537
 2019-11-19 14:45:39 +01:00
Sebastian Sumpf
417dd59b22 libports: gcov for ARMv8 
issue #3537
 2019-11-19 14:45:39 +01:00
Sebastian Sumpf
3024720656 ports: update make to version 4.2.1 
Required for ARMv8 support

issue #3537
 2019-11-19 14:45:39 +01:00
Sebastian Sumpf
f23eab735b ports: binutils target for ARMv8 
issue #3537
 2019-11-19 14:45:39 +01:00
Sebastian Sumpf
112c32eb54 ports: noux-pkg/gcc_arm_64 
Added ARMv8 as a native gcc target. This requires a rebuild of the tool
chain.

issue #3537
 2019-11-19 14:45:39 +01:00
Sebastian Sumpf
3103ce1fa8 libports: update mpc to 1.1.0 
issue #3537
 2019-11-19 14:45:39 +01:00
Sebastian Sumpf
db18fc42fe libports: update mpfr to 4.0.2 
issue #3537
 2019-11-19 14:45:39 +01:00
Sebastian Sumpf
d28fe9e938 libports: enable gmp 6.1.2 for ARMv8 
issue #3537
 2019-11-19 14:45:39 +01:00
Sebastian Sumpf
31a035a907 libports: update gmp from 4.3.2 to 6.1.2 
issue #3537
 2019-11-19 14:45:39 +01:00
Norman Feske
91412c6c52 libc: trigger SIGCHLD for when forked child exits 2019-11-19 14:45:39 +01:00
Norman Feske
068324536c libc: eliminate atexit handler for 'Rtc' 
We never want to destruct the 'Rtc' because other atexit handlers may
depend on it (e.g., for updating modification times when closing files).
 2019-11-19 14:45:38 +01:00
Norman Feske
bb6eb0f6ea libc: local signal delivery via 'kill' 
This patch adds the ability to call 'kill' with the own PID to trigger
the execution of the handler of the specified POSIX signal. This is used
by 'bash', e.g., when cancelling the input of a command via control-c.

Related to issue #3546
 2019-11-19 14:45:38 +01:00
Norman Feske
c1012e6a45 libc: translate terminal user interrupts to SIGINT 
Issue #3546
 2019-11-19 14:45:38 +01:00
Norman Feske
23d21d77e9 vfs/terminal: detect user interrupts (control-c) 
This patch equips the VFS terminal file system with the ability to
detect user interrupts in the incoming data stream, and propagate this
information via the new pseudo file '.terminal/interrupts'. Each time,
the user presses control-c in the terminal, the value stored in this
pseudo file is increased. Thereby, a VFS client can watch this file to
get notified about the occurrences of user interrupts.

Fixes #3546
 2019-11-19 14:45:38 +01:00
Norman Feske
8c44b17e86 libc: shebang handling for execve 
Fixes #3545
 2019-11-19 14:45:38 +01:00
Norman Feske
636e0f6444 libc: cancel select when POSIX signal occurs 
With this patch, Vim running via the 'bash.run' script becomes able to
adopt itself to changed window dimensions.

Issue #3544
 2019-11-19 14:45:36 +01:00
Josef Söntgen
fafa409cf9 block_tester: use dynamic shared scratch buffer 
The scratch buffer is now allocated from the heap and is shared
between the test as they are executed in a serial fashion. This
change saves memory as the test are constructed at start-up.

Fixes #3539.
 2019-11-19 14:44:02 +01:00
Josef Söntgen
dbecceec09 expanding_reporter: configure initial buffer size 
Fixes #3543.
 2019-11-19 14:44:02 +01:00
Josef Söntgen
60f390ddf8 noux: prevent accessing invalid leaf path in open 
Fixes #3542.
 2019-11-19 14:44:02 +01:00
Josef Söntgen
c79ebc93a2 libc_noux: try read again on EAGAIN/EINTR 
Issue #3541.
 2019-11-19 14:44:02 +01:00
Josef Söntgen
55ab694d79 libc_noux: leave write loop on error 
Fixes #3541.
 2019-11-19 14:44:02 +01:00
Josef Söntgen
f5c5479faa ram_block: use Number_of_bytes to parse block size 
Using this helper rather than parsing as size_t directly allows use to
use size suffixes, e.g. K, M, G.

Fixes #3540.
 2019-11-19 14:44:02 +01:00
Josef Söntgen
ba9b612c4f libc_noux: close fd on failed O_TRUNC attempt 
Fixes #3524.
 2019-11-19 14:44:02 +01:00
Norman Feske
7b0771659e libc: trigger SIGWINCH by watching .terminal/info 
Issue #3544
 2019-11-19 14:43:43 +01:00
Norman Feske
e9762ee25f vfs: 'Rtc_file_system::close' for watch handle 
The default implementation of this method throws an exception.
 2019-11-19 14:42:23 +01:00
Norman Feske
7549189f88 vfs: watch support for Readonly_value_file_system 
This patch adds support for watch notifications for the
'Readonly_value_file_system', which is often used by VFS plugins to
implement pseudo files. It thereby enables VFS clients to respond to
VFS-plugin events (think of terminal resize) dynamically.

Fixes #3523
 2019-11-19 14:42:23 +01:00
Oleg Girko
a5bc031cca genode_org: Fix download URL. 
As genode.org site is now protected by TLS and unencrypted site
is redirected to enctypted one, download URL should be modified
to reflect this change.

Without this change, wget downloads just index.html file.

Signed-off-by: Oleg Girko <ol@infoserver.lv>

Fixes #3538
 2019-11-19 14:42:23 +01:00
Oleg Girko
5cd684997a sntp_client: No need to always limit this scenario to x86. 
It runs pretty well on Raspberry Pi, for example.

Leaving this scenario limited to x86 for autopilot runs only.

Signed-off-by: Oleg Girko <ol@infoserver.lv>

Fixes #3536
 2019-11-19 14:42:23 +01:00
Christian Helmuth
93d3a0848a Remove unused libm/posix.lib.so from lwip_lx.run 
Prevents integration failures like follows if the libraries were not
built already in the build directory.

  Missing boot modules: libm.lib.so posix.lib.so
  Makefile:323: recipe for target 'run/lwip_lx' failed
 2019-11-19 14:42:23 +01:00
Christian Helmuth
f1b1dd26cf run: tweak CRLF translation in log scripts 
Ideally, I'd like to disable expects's translation here, which does not
work. The current setting is the result of testing all combinations.
 2019-11-19 14:42:23 +01:00
Christian Helmuth
cd5e906bd0 imx: do not convert LF to CRLF in UART driver 
This is done on higher level already.
 2019-11-19 14:42:23 +01:00
Oleg Girko
c589660182 Replace ENV make variable with MKENV. 
ENV variable conflicts with Environment Modules.

Signed-off-by: Oleg Girko <ol@infoserver.lv>

Fixes #3535
 2019-11-19 14:42:23 +01:00
Stefan Kalkowski
8a8aa85726 hw: initialize iomux, ccm and gpc for i.MX8 EVK 
Fix #3534
 2019-11-19 14:42:23 +01:00
Stefan Kalkowski
105b2c9b7a hw: fix gicv3 implementation of clear/set regs 
In addition use uniformly enums for interupt count in register declarations.

Fix #3532
 2019-11-19 14:42:23 +01:00
Stefan Kalkowski
f6435d91fc hw: turn Kernel_object into Genode::Constructible 
Fix #3531
 2019-11-19 14:42:23 +01:00
Stefan Kalkowski
3e3fb63863 hw: enable Genode::raw for bootstrap 
Ref #3531
 2019-11-19 14:42:23 +01:00
Oleg Girko
4007cee852 Fix MAC address printing in dde_linux USB net drivers. 
Passing array of unsigned chars to Genode::log() function
makes it converted to void pointer, resulting in printing its address.

Wrapping this array into Genode::Cstring solves this problem
and makes it being printed properly as zero-terminaled string.

Signed-off-by: Oleg Girko <ol@infoserver.lv>

Fixes #3530
 2019-11-19 14:42:23 +01:00
Christian Helmuth
b622a5a788 vbox: use RTThreadCreate for periodic_gip thread 
This prevents the following error message.

  [init -> vbox1] Error: pthread_self() called from alien thread named 'periodic_gip'
 2019-11-19 14:42:23 +01:00
Christian Helmuth
ec9e40695d vbox: use PIIX3 instead of ICH9 in VM config 
Prevents #3533
 2019-11-19 14:42:23 +01:00
Martin Stein
0ad0153626 nic_router: fix broken uplink label on re-config 
Uplink labels were stored in the uplink objects as reference. As uplink objects
may be re-used on re-configuration to avoid unnecessary closing and re-opening
of NIC sessions, the label reference could get invalid as the object that
stored the label content didn't get re-used. Thus, the label is now copied to
the uplink object.

Fixes #3529
 2019-11-19 14:42:23 +01:00
Martin Stein
cd37bff514 nic_router: do not report while re-configuring 
Fixes #3528
 2019-11-19 14:42:23 +01:00
Martin Stein
27c2a66bbd nic_router: report link state 
This adds two new boolean attributes to the <report> tag of the NIC router
configuration 'link_state' and 'link_state_triggers'. The former decides
whether to report the link state of each NIC interface (downlink, uplinks) at
the NIC router. The other decides whether to trigger reporting each time the
link state of an interface changes.

Fixes #3527
 2019-11-19 14:42:23 +01:00
Martin Stein
58247737fd nic_router: free MAC on session-creation exception 
If a MAC address was once allocated for a downlink and during the further
creation of the downlink an exception caused the creation to be aborted, the
NIC router didn't free the MAC address again.

Ref #3525
 2019-11-19 14:42:23 +01:00
Martin Stein
ebcca179ed nic_router: fix 'catch' paths in session creation 
The handling of exceptions during session creation implies freeing and
detaching the RAM DS behind the session env and session component. But this
was done using the session env located in the same RAM DS, which must
lead to problems. Now, the session env from the RAM DS is copied to the stack
and the operations are done on this temporary instance instead.

Ref #3525
 2019-11-19 14:42:22 +01:00
Martin Stein
60d37f690c packet_stream_*: fix missing dissolve on exception 
When the construction of a member of Packet_stream_*::Rpc_object after
the _cap member threw an exception, the object was not dissolved from
the entrypoint although the Rpc_object vanished at this point. This was
because the call to 'manage()' happened in the initializer list (for the
_cap member instantiation). The destruction of the _cap member then did
not dissolve the object.

This first fix moves the call to 'manage()' into the constructor body
after the instantiation of all other members. A more sophisticated fix
would use some kind of 'Managed_object' life-time guard that manages an
object on construction and dissolves on destruction.

Ref #3525
 2019-11-19 14:42:22 +01:00
Martin Stein
0ed5655086 nic_router: do DHCP when re-config to dynamic IP 
When re-configuring a domain from using a static IP config to using a dynamic
IP config, attached interfaces didn't react with doing DHCP discover. This bug
is fixed now and the nic_router_flood test was adapted in a way that it tests
the described behavior.

Fix #3518
 2019-11-19 14:42:22 +01:00
Stefan Kalkowski
87a6368ba1 hw: implement multi-processor support for rpi3 
Fix #3522
 2019-11-19 14:42:22 +01:00
Stefan Kalkowski
1cbd77c806 hw: implement multi-processor support for i.MX8 
Fix #3520
 2019-11-19 14:42:22 +01:00
Stefan Kalkowski
e3f82b09d7 hw: instantiate pic object per cpu 
Ref #3520
 2019-11-19 14:42:22 +01:00
Christian Helmuth
d4a3db22bd solo5: migrate snapshot archive to genode.org 2019-11-19 14:42:22 +01:00
Christian Helmuth
43f28e0451 stdcxx: add more missing symbols to ABI 
Fixes #3506
 2019-11-19 14:42:22 +01:00
Norman Feske
d516515c7a vfs/terminal: provide meta info as pseudo files 
Issue #3519
Fixes #3521
 2019-11-19 14:39:09 +01:00
Norman Feske
7ac32ea60c libc: support for ioctls via ioctl directory 
This patch introduces a new scheme of handling ioctl operations that
maps ioctls to pseudo-file accesses, similar to how the libc maps socket
calls to socket-fs operations.

A device file can be accompanied with a (hidden) directory that is named
after the device file and hosts pseudo files for triggering the various
device operations. For example, for accessing a terminal, the directory
structure looks like this:

  /dev/terminal
  /dev/.terminal/info

The 'info' file contains device information in XML format. The type of
the XML node corresponds to the device type. E.g., If the libc receives
a 'TIOCGWINSZ' ioctl for /dev/terminal, it reads the content of
/dev/.terminal/info to obtain the terminal-size information. In this
case, the 'info' file looks as follows:

  <terminal rows="25" columns="80/>

Following this scheme, VFS plugins can support ioctl operations by
providing an ioctl directory in addition to the actual device file.

Internally, the mechanism uses the 'os/vfs.h' API to access pseudo
files. Hence, we need to propagate the Vfs::Env to 'vfs_plugin.cc' to
create an instance of a 'Directory' for the root for the VFS.

Issue #3519
 2019-11-19 14:39:09 +01:00
Christian Prochaska
07a40d028a libc: fix and cleanup pthread mutexes 
Issue #3503
Fixes #3504
 2019-11-19 14:23:57 +01:00
Christian Prochaska
a47adecdcd test-pthread: add mutex tests 
Add tests for recursive and errorcheck type mutexes.

Fixes #3503
 2019-11-19 14:23:57 +01:00
Christian Helmuth
355d94f5df Use NTP server from pool.ntp.org in sntp_client test 2019-11-19 14:23:57 +01:00
Pirmin Duss
c0789a6c0e depot_deploy: status report 
The idea is, that other components may know, when a valid
init.config is available.

Issue #3482
 2019-11-19 14:23:57 +01:00
Norman Feske
530144b040 doc/challenges.txt: add Puppet topic 2019-11-19 14:23:56 +01:00
Norman Feske
c85bc38802 Move include/gems/vfs.h to include/os/vfs.h 
Fixes #3515
 2019-11-19 14:23:56 +01:00
Norman Feske
a8dd7dd2fa Remove rump_fs server component 
This patch replaces the use of the rump_fs server by the VFS server with
the Rump VFS plugin.

Issue #3512
 2019-11-19 14:23:56 +01:00
Norman Feske
6e86d6d699 Remove server/fatfs_fs and libc_fatfs plugin 
Issue #3512
 2019-11-19 14:23:56 +01:00
Norman Feske
2954abb58a Simplify qt5_textedit.run 
The run script used to be the only user of the fatfs_fs server, which
we're going to remove. This patch removes the components for accessing a
real storage device and file system from the run script. The new version
just uses ram_fs.

Issue #3512
 2019-11-19 14:23:56 +01:00
Norman Feske
5bb366513b Remove os/src/app/cli_monitor 
Nowadays, we use standard command-line tools like vim to edit init
configurations dynamically, which alleviates the need for a custom CLI.
The CLI-monitor component was too limited for use cases like Sculpt
anyway.

The patch also removes the ancient (and untested for long time)
terminal_mux.run script, which used to be the only remaining user of the
CLI monitor.

Issue #3512
 2019-11-19 14:23:56 +01:00
Norman Feske
4bcc75365c Remove gems/src/app/launcher 
The graphical launcher remained unused for a few years now. It is not
suitable for systems as flexible as Sculpt OS.

Issue #3512
 2019-11-19 14:23:56 +01:00
Emery Hemingway
288f79270d test/vfs_stress: annotate switch label fall-throughs 
This test was written for speed, not correctness.

Fixes #3516
 2019-11-19 14:23:56 +01:00
Christian Prochaska
3c62a33a25 qt5: add NEON-related files and symbols to qt5_gui lib 
Fixes #3517
 2019-11-19 14:23:56 +01:00
Piotr Tworek
80a84bef26 Fix ram session quota accounting in lan9118 
This driver manually checks if the RAM quota is big enough for the
Lan9118 nic session component. The problem is Root_component::_create
from which Root_component::_create_session gets called does already
check this. No need to account for it twice.

Fixes #3514
 2019-11-19 14:23:56 +01:00
Johannes Kliemann
022fac0d37 dde_zircon: use fork by jklmnn 
Fixes #3513
 2019-11-19 14:23:56 +01:00
Norman Feske
5ab1505d43 file system: enhanced file status info 
This patch extends the 'File_system::Status',
'File_system::Directory_entry', and the related 'Vfs' types with
the following additional information:

- Distinction between continuous and transactional files (Node_type)
  (issue #3507)
- Readable, writeable, and executable attributes (Node_rwx),
  replacing the former 'mode' bits
  (issue #3030)

The types 'Node_rwx', 'Node_type' are defined twice,
once for the VFS (vfs/types.h) and once for the 'File_system'
session (file_system_session/file_system_session.h).
Similarly, there is a direct correspondance between
'Vfs::Directory_service::Dirent' and 'File_system::Directory_entry'.

This duplication of types follows the existing pattern of keeping the
VFS and file-system session independent from each other.
 2019-11-19 14:23:56 +01:00
Alexander Boettcher
1297a8fb57 trace: skip subject_id 0 in subject id allocator 
Issue #3510
 2019-11-19 14:23:55 +01:00
Christian Helmuth
732215a83f libc: limit fcntl(F_SETFL) to file-status flags 
This fixes unintended (and unpermitted) changes of O_ACCMODE bits.
 2019-11-19 14:23:55 +01:00
Christian Helmuth
e11addec7d Cleanup vfs dev directory in vbox5_genode_usb_hid 2019-11-19 14:23:55 +01:00
Christian Helmuth
2166a4b17f dde_linux: prevent warning about 'register' keyword 2019-11-19 14:23:55 +01:00
Christian Helmuth
291587f545 libc: honor poll() event flags POLLRDNORM etc. 
Fixes empty read file-descriptor sets in fetchurl_lwip/lxip.

Issue #3499
 2019-11-19 14:23:55 +01:00
Christian Helmuth
97df705e53 dde_rump: eliminate global ctors 2019-11-19 14:23:55 +01:00
Josef Söntgen
e281174dae libc: bring back old 'poll()' emulation 
Issue #3499
 2019-11-19 14:23:55 +01:00
Alexander Boettcher
ed73feddc5 vbox5: react on host size changes immediately 
The guest may take some time to adjust nevertheless.

Fixes #3498
 2019-11-19 14:23:55 +01:00
Josef Söntgen
c5706e8f4a wifi_drv: enable loading of 5000 series FW 
Fixes #3502.
 2019-11-19 14:23:55 +01:00
Christian Helmuth
1782c6be79 qt5: rework keyboard handling 
Fixes #3483
 2019-11-19 14:23:54 +01:00
Christian Helmuth
cccfd0719d qt5: prevent deprecated warnings 
Issue #3483
 2019-11-19 14:23:54 +01:00
Christian Helmuth
edc9545229 input: move control-modifier handling into clients 
Moving the handling into the input-session clients enables more
sophisticated implementations (like Qt5) to apply key-symbol based
handling of those modifiers like correct CTRL-A with QWERTY and AZERTY
layouts and distinction of CTRL-J and Return.

Issue #3483
 2019-11-19 14:23:54 +01:00
Christian Helmuth
cc611834c9 input: repeat events require valid codepoints 
Issue #3483
 2019-11-19 14:23:54 +01:00
Christian Helmuth
bbd27a54d3 input_filter: fix de_de/fr_fr character generators 
Issue #3483
 2019-11-19 14:23:54 +01:00
Christian Helmuth
5a06751242 Increase input_filter RAM quotas to 1280K 
Issue #3483
 2019-11-19 14:23:54 +01:00
Christian Helmuth
6df8b44616 platform/x86: support 64-bit base address registers 
The API still exports 32-bit address and size values only, which works
as the actual MMIO resources are allocated in platform_drv internally.

Fixes #3494
 2019-11-19 14:23:54 +01:00
Christian Helmuth
e0af9c2d8b platform_drv.inc: reduce acpi_drv caps to 350 
This value equals the configuration in drivers_managed-pc, i.e., Sculpt.
 2019-11-19 14:23:54 +01:00
Christian Helmuth
11a7ac0536 acpi_drv: reduce capability quota requirements 
Map ACPI tables entirely as one dataspaces (not page-by-page). Note, the
current approach does only merge overlapping but not consecutive I/O
memory regions, which would reduce the amount of capabilities required
even further.

Fixes #3495
 2019-11-19 14:23:54 +01:00
Christian Prochaska
5c25e0bdb0 heap: fix typo in alloc(0) diagnostic message 
Fixes #3501
 2019-11-19 14:19:34 +01:00
Norman Feske
222f214341 extract.run: suppress rtc warnings 2019-11-19 14:19:34 +01:00
Norman Feske
eefe91ee41 depot_download_manager: no mtime update in extract 
With the added modification-time support in the libc, the extract tool
requires a timer session, which is not plausible for the purpose of the
program.

This behavior stems from the fact that the libc implicitly writes the
mtime when closing a written file. For this update, it implicitly calls
'clock_gettime', which in turn initializes the timer subsystem within
the libc (creating a timer session).

For the extract tool, the implicitly updated mtime is useless because
the extract tool overwrites this modification time with the mtime stored
in the archive anyway. However, the dependency from a timer service
remains.

This patch explicitly disables the libc's implicit updating of the
file-modification when closing a written file.

Issue #1784
 2019-11-19 14:19:34 +01:00
Norman Feske
979d823d85 libc: make mtime update configurable 
By specifying <libc update_mtime="no"...>, the modification-time update
on VFS-sync operations (as issued whenever a written file is closed)
can explicitly be disabled.

Issue #1784
 2019-11-19 14:19:34 +01:00
Christian Helmuth
76438a3f85 dde_linux: refactor usb arch-dependent includes 
Also added arm_64/arm_v8, which fixes building board-independent usb_hid
etc.
 2019-11-19 14:19:34 +01:00
Christian Helmuth
b0271ae5e1 Fix hard-disk image creation in vfs server tests 
The hard-disk image must be created before boot-image creation as it is
used as boot module by variants of the test (use_ram_block=true).
 2019-11-19 14:17:30 +01:00
Norman Feske
6a063364da fs_rom: adaptation to added file-modification time 
Issue #1784
 2019-11-19 14:17:30 +01:00
Josef Söntgen
99b632f86c server/vfs: add modification time 
Issue #1784.
 2019-11-19 14:17:30 +01:00
Josef Söntgen
180f9e6384 libc: handle modified files 
Issue #1784.
 2019-11-19 14:17:30 +01:00
Josef Söntgen
94b63924ed noux: add time modification 
... and set initial time by using RTC session.

Up to now Noux used a monotic clock whose initial start value always
was '0' (which correlates to the start of the UNIX epoch) to provide
a notion of time. In addition it is now possible to use the RTC
session to set the initial value for use cases where having a proper
real-world time matters.

To use the RTC session the 'rtc' attribute of the '<config>' node
must be set to 'yes'. Thereby the session becomes a mandatory
dependency as Noux will not start without it.

Issue #1784
 2019-11-19 14:17:30 +01:00
Josef Söntgen
d0bf6d2b52 libc: add modification time 
Issue #1784.
 2019-11-19 14:17:30 +01:00
Josef Söntgen
9a82bbb54d vfs: add modification time member to directory_service 
Issue #1784.
 2019-11-19 14:17:30 +01:00
Josef Söntgen
400039e1b6 rump_fs: add modification time 
Issue #1784.
 2019-11-19 14:17:30 +01:00
Josef Söntgen
2e5166efd7 lx_fs: add modification time 
Issue #1784.
 2019-11-19 14:17:30 +01:00
Josef Söntgen
2ec3aaf639 file_system_session: add modification time 
* add modification_time member
* add WRITE_TIMESTAMP packet
 2019-11-19 14:17:30 +01:00
Norman Feske
ab5187d673 libc: resolve symlinks in execve 
This patch complements the commit "libc: execve" with the ability to
execute files stored at arbitrary sub directories of the file system.

Issue #3481
Issue #3500
 2019-11-19 14:17:29 +01:00
Norman Feske
697d496093 ldso: don't strip leading path from ROM name 
This patch removes ldso's builtin policy of removing any path elements
prepending the ROM module name. Instead, the ROM name is used as is.
This clears the way to access different ROM modules that share the same
name but are stored at different directories behind an fs_rom (e.g.,
/bin/bash vs. /usr/local/bin/bash).

Issue #3500
 2019-11-19 14:17:29 +01:00
Norman Feske
a17c5e30b7 mk/gnu_build.mk: avoid absolute lib paths 
This patch changes the way how libraries are specified at the linker
command line from /abs/path/to/library.lib.so

Issue #3500
 2019-11-19 14:17:29 +01:00
Christian Helmuth
8d6285927b tools.conf: fix check for arm_64 
Before arm matched first resulting in the wrong tool-chain binaries
being used.
 2019-11-19 14:17:29 +01:00
Norman Feske
90a91f3536 dde_linux: eliminate global ctors in lx_kit 
As the lx_kit library is used by the vfs_lxip plugin, it must not
contain any static global constructor.

Related to issue #3487
 2019-11-19 14:17:29 +01:00
Norman Feske
c8b7710e5d libc: improve dup/dup2 in vfs_plugin 
This patch replaces the naive dup2 implementation (that merely
duplicated the context pointer) by the replication of the original
FD state by re-opening the same file with the same flags and seek
position. This prevents a potential double release of the VFS handle
(the FD context). It also implements 'dup'.

Fixes #3505
Fixes #3477
 2019-11-19 14:10:55 +01:00
Norman Feske
59c60b8031 test-libc: test double dup2, dup 
Issue #3477
Issue #3505
 2019-11-19 14:10:55 +01:00
Norman Feske
9500e8b6e1 lwip: remove use of global constructor 
Related to issue #3487
 2019-11-19 14:10:55 +01:00
Norman Feske
e0ee56275e vfs_lwip: fix switch fall-through 
Pointed out by a compiler warning.
 2019-11-19 14:10:55 +01:00
Norman Feske
418ac4c560 libc: remove global watch() function 
This patch replaces the function with a 'Watch' interface to be
explicitly passed to the caller (currently only time.cc).

Issue #3497
 2019-11-19 14:10:55 +01:00
Norman Feske
5f5d709c07 libc: remove global libc_config() 
Issue #3497
 2019-11-19 14:10:55 +01:00
Norman Feske
89a38723bd libc: reimplement passwd handling 
- Eliminate call of global libc_config()
- Remove dynamic memory allocation, const cast
- Prepare for moving the state from compilation unit to header
- Fix run/libc_getpwent.run

Issue #3497
 2019-11-19 14:10:55 +01:00
Norman Feske
648bcd1505 libc: unify use of namespaces 
This patch unifies the patterns of using the 'Genode' and 'Libc'
namespaces.

Types defined in the 'internal/' headers reside in the 'Libc'
namespace. The code in the headers does not need to use the
'Libc::' prefix.

Compilation units import the 'Libc' namespace after the definition of
local types. Local types reside in the 'Libc' namespace (and should
eventually move to an 'internal/' header).

Since the 'Libc' namespace imports the 'Genode' namespace, there is
no need to use the 'Genode::' prefix. Consequently, code in the
compilation units rarely need to qualify the 'Genode' or 'Libc'
namespaces.

There are a few cases where the 'Libc', the 'Genode', and the global
(libc) namespaces are ambigious. In these cases, an explicit
clarification is needed:

- 'Genode::Allocator' differs from 'Libc::Allocator'.
- 'Genode::Env' differs from 'Libc::Env'.
- Genode's string functions (strcmp, memcpy, strcpy) conflict
  with the names of the (global) libc functions.
- There exist both 'Genode::uint64_t' and the libc'c 'uint64_t'.

Issue #3497
 2019-11-19 14:10:55 +01:00
Norman Feske
bf92232698 libc: split task.cc into multiple files 
This patch is the first step of re-organizing the internal structure of
the libc. The original version involved many direct calls of global
functions (often with side effects) across compilation units, which
made the control flow (e.g., the initialization sequence) hard to
follow.

The new version replaces those ad-hoc interactions with dedicated
interfaces (like suspend.h, resume.h, select.h, current_time.h). The
underlying facilities are provided by the central Libc::Kernel and
selectively propagated to the various compilation units. The latter is
done by a sequence of 'init_*' calls, which eventually will be replaced
by constructor calls.

The addition of new headers increases the chance for name clashes with
existing (public) headers. To disambiguate libc-internal header files
from public headers, this patch moves the former into a new 'internal/'
subdirectory. This makes the include directives easier to follow and the
libc's source-tree structure more tidy.

There are still a few legacies left, which cannot easily be removed
right now (e.g., because noux relies on them). However, the patch moves
those bad apples to legacy.h and legacy.cc, which highlights the
deprecation of those functions.

Issue #3497
 2019-11-19 14:10:55 +01:00
Norman Feske
aec1178ab1 libc: simplify initialization sequence 
This patch reverts the interim patch "libc: init malloc_heap before
VFS", which is no longer needed with the solved issue #3496 and
issue #3487.
 2019-09-20 14:16:07 +02:00
Norman Feske
efe7f5172d ldso: prevent premature execution of ctors 
Fixes #3487
 2019-09-20 14:16:05 +02:00
Norman Feske
0aedabd245 libc: remove global constructors 
- readv_writev: move 'rw_lock' instance into a function scope,
  constructing the instance on the first access.
- select: move 'select_cb_list' instance into function scope.
- thread: move 'key_list_lock' and 'keys' into function scope.
- rwlock, semaphore, socket_fs_plugin, thread, thread_create:
  instantiate 'Libc::Allocator' per use, alleviating the need for a
  global instance.

Issue #3496
 2019-09-20 14:16:01 +02:00
Christian Helmuth
4a7b0e99a6 depot: update recipe hashes 2019-09-20 14:14:16 +02:00
Norman Feske
614f7fa56a mesa.run: omit depot content from boot modules 2019-09-17 14:38:07 +02:00
Norman Feske
6d230134cb doc: update challenges 2019-09-13 12:14:19 +02:00
Pirmin Duss
d953030c0e lx_fs: respect writeable session argument 
Fixes #3492
 2019-09-12 12:28:41 +02:00
Alexander Boettcher
6fe80c3cc7 sculpt: add lock screen/GUI support 
Fixes #3491
 2019-09-09 15:25:45 +02:00
Christian Prochaska
afa0e26a6a qt5: update virtual keyboard example to Qt 5.13 
Fixes #3490
 2019-09-09 15:08:13 +02:00
Christian Prochaska
1d3bbde70a qt5: add 'lupdate' and 'lrelease' to host tools 
Fixes #1736
 2019-09-09 15:05:50 +02:00
Alexander Boettcher
6f0c6501f2 acpica: catch Io_mem exceptions 
Issue #2998
 2019-09-09 15:04:53 +02:00
Christian Helmuth
e448022f23 Fix hard-disk image creation in libc_fatfs 
The hard-disk image must be created before boot-image creation as it is
used as boot module by variants of the test (use_ram_block=true).
 2019-09-09 14:58:25 +02:00
Christian Helmuth
664d858e63 Fix sntp_client test for hw/muen 2019-09-09 12:15:07 +02:00
Christian Helmuth
ede009edf9 Fix expected output in USB HID tests 
Invalid characters in press events are no longer 0 but 0xfffe since

  a97b8043b5 utf8: non-character U+fffe as invalid codepoint

Issue #3483
 2019-09-09 11:35:42 +02:00
Christian Helmuth
08aa7d310a Increase stack size in thread test for arm_v8a 
It seems throwing an exception on arm_v8a needs a significant amount of
stack space (> 2048 bytes).
 2019-09-03 12:12:46 +02:00
Christian Helmuth
19bf0fdeb8 xkb2ifcfg: detect and log initialization errors 2019-09-03 09:00:31 +02:00
Emery Hemingway
36111a2edf Libc: implement getifaddrs 
Implement getifaddrs and freeifaddrs within the libc using socket
control files at the VFS. Add an "address" and "netmask" file to the
lwIP plugin.

Only a single IPv4 address is initially supported, and the broadcast
address returned will never be valid.

Fixes #3439
 2019-09-02 16:36:07 +02:00
Christian Prochaska
ab017607a2 tool chain: TLS support 
Fixes #3418
 2019-09-02 16:29:33 +02:00
Christian Helmuth
2c47cd5c94 version: 19.08 2019-08-28 14:36:57 +02:00
Norman Feske
44ce1b37c1 News item for version 19.08 2019-08-28 14:36:57 +02:00
Norman Feske
9a52a93c24 Release notes for version 19.08 2019-08-28 14:36:57 +02:00
Christian Helmuth
b2c59576ae depot: update recipe hashes 2019-08-28 14:36:56 +02:00
Norman Feske
7f8fe00cdc drivers_managed-pc: route for smbios_table report 2019-08-28 14:22:33 +02:00
Norman Feske
c0e8336e98 acpi_drv: report SMBIOS after ACPI info 
This order of execution reduces the boot time of Sculpt.
 2019-08-28 14:22:33 +02:00
Norman Feske
155e214a69 libc: init malloc_heap before VFS 
This is an interim fix for issue #3487.
 2019-08-28 14:22:33 +02:00
Christian Helmuth
4911acedf5 sculpt: version 19.08 2019-08-28 14:22:33 +02:00
Christian Helmuth
01bd87e90b tool/port: re-enable check_tool in install.mk 
The check was accidentally in

  66d44289e1 tool/ports: streamline hash tool usage

because $(call VAR, ...) just expands to nothing without errors if VAR
is undefined.
 2019-08-28 14:22:33 +02:00
Christian Prochaska
65f402807f qt5: update to version 5.13.0 
Fixes #3485
 2019-08-28 14:22:33 +02:00
Christian Prochaska
d00cfd7cff stdcxx: add missing symbol for qt5 update 
Fixes #3486
 2019-08-28 14:22:33 +02:00
Christian Helmuth
701b1d41e8 input_filter: update character generators 
This adds complete character-generator configurations for English (US),
German (Germany and Switzerland), and French (France and Switzerland).
The configs are manually amended and stripped-down versions of
xkb2ifcfg generated configs.

Issue #3483
 2019-08-28 14:22:33 +02:00
Sebastian Sumpf
8ae5c906d0 noux_tool_chain_auto: use absolute paths for command 
issue #3481
 2019-08-28 14:22:32 +02:00
Christian Helmuth
43a8118d2e tool: generate input_filter config from XKB 
Issue #3483
 2019-08-28 14:22:32 +02:00
Christian Helmuth
ca850c787f input_filter: dead-key sequence support 
Issue #3483
 2019-08-28 14:22:32 +02:00
Christian Helmuth
4491c070be input: fix Codepoint::INVALID in Press event 
The Press event is actually a Press_char event with a default codepoint.
The default codepoint is now

  Codepoint { Codepoint::INVALID } /* value 0xfffe */

in contrast to

  Codepoint { Input::Event::INVALID } /* value 0 */

Issue #3483
 2019-08-28 14:22:32 +02:00
Christian Helmuth
a97b8043b5 utf8: non-character U+fffe as invalid codepoint 
Unicode non-characters [1] are guaranteed to never be used for a
character. The formerly used U+fffd however is a valid character - the
replacement character [2] correctly displayed by Qt5 as <?>.

[1] https://en.wikipedia.org/wiki/Universal_Character_Set_characters#Non-characters
[2] https://en.wikipedia.org/wiki/Specials_(Unicode_block)#Replacement_character

Issue #3483
 2019-08-28 14:22:32 +02:00
Christian Helmuth
4967166811 run: exit if boot modules are missing on linux 2019-08-28 14:22:32 +02:00
Martin Stein
cc2828cf3a sntp_client: improve README, clean source code 
Fixes #3448
 2019-08-28 14:21:36 +02:00
Norman Feske
1dd68ce04b Adjust quotas for seL4 2019-08-28 14:19:45 +02:00
Norman Feske
23f3112e3e ports: bash.run script using the libc fork/execve 
Issue #3481
 2019-08-28 14:19:45 +02:00
Norman Feske
6894ced63b libc: execve 
This patch implements 'execve' in Genode's libc.

The mechanism relies on the dynamic linker's ability to replace the
loaded binary while keeping crucial libraries - in particular the libc -
intact. The state outside the libc is wiped. For this reason, all libc
internal state needed beyond the 'execve' call must be allocated on a
heap separate from the application-owned malloc heap. E.g.,
libc-internal file-descriptor objects must not be allocated or refer to
any memory object allocated from the malloc heap.

Issue #3481
 2019-08-28 14:19:45 +02:00
Alexander Senier
2a3cebdd6e libc: Allow private writable mappings 
Fixes #3480
 2019-08-28 14:18:45 +02:00
Norman Feske
66d5359d75 ldso: heuristics for libs needed during execve 
The allocation of regions within the linker area is normally left to the
best-fit 'Allocator_avl', which happens to populate the linker area
starting with the binary followed by all loaded libraried with no gaps
in between.

When replacing the binary during execve, however, we need to ensure that
the new binary does not conflict with any library that stays resident
during execve. This patch tweaks the linker's region allocation scheme
such that these libraries are placed at the end of the linker area.

Issue #3481
 2019-08-28 14:18:45 +02:00
Norman Feske
fa48054959 ldso: Dynamic_linker::respawn mechanism 
This patch extends the interface of the dynamic linker with the ability
to replace the running binary executable by another one. It is
designated for the implementation of execve. The interface consists of
two new functions.

'Dynamic_linker::keep' marks the specified shared object as unloadable.
This can be used to pin a set of libraries (i.e., the libc) within the
local address space while replacing the binary and other higher-level
libraries.

'Dynamic_linker::respawn' unloads the current binary, loads the one
specifed as first argument, and looks up the entry point symbol of the
new binary, which would be "main" for POSIX programs.

In addition to implementing the new interface, the patch adjusts the
linker at various places that previously assumed the binary to be
constant over runtime.

Issue #3481
 2019-08-28 14:18:45 +02:00
Norman Feske
808ff3714e rust: don't export cxx syms from libunwind-rust 2019-08-28 14:18:45 +02:00
Norman Feske
a98f78afd9 ldso: avoid 'dd' output in build log 
This patch is a follow-up commit for "ld: load dynamic linker at static
address on Linux". It suppresses the stderr output of 'dd' when marking
the ELF binary as executable.

Issue #3479
 2019-08-28 14:18:45 +02:00
Sebastian Sumpf
4c14af4d8a noux-pkg: enable noux packages for ARMv8 
* add -fPIC
* disable 'stdbuf' for coreutils
* works for bash as well
 2019-08-28 14:18:45 +02:00
Norman Feske
817ff6ca68 posix: improve config error diagnostics 2019-08-28 14:18:45 +02:00
Norman Feske
3c6fe6e741 ldso: statically allocate initial heap block 
This patch enables the fork.run script to run on base-linux. It should
be regarded as an interim solution, however, because the randomization
performed by the Linux kernel may still - by chance - produce a
situation where one of the libc's malloc heap regions intersects with
another dataspace dynamically attached to the child.

The better solution would be to make the 'Region_map_mmap'
implementation not depend on the kernel's allocation policy by using a
locally implemented allocator.

Issue #3478
 2019-08-28 14:18:45 +02:00
Norman Feske
bb5827b4e3 libc: fork, getpid, and wait4 
This patch complements the C runtime with support for fork, getpid, and
wait4 (and its cousin 'waitpid').

Fixes #3478
 2019-08-28 14:18:45 +02:00
Norman Feske
581785a48f Extend fork test 
This patch extends the fork test with explicit checks for the cloned
content of the heap and RW segment as well as the seek position of an
open file descriptor. It adds the new libports/run/fork.run script
that exercises the fork mechanism implemented by the libc. It is based
on noux_fork.run, which tests the mechansim provided by noux. The
test program has been moved from ports to libports.

Issue #3478
 2019-08-28 14:18:44 +02:00
Norman Feske
65f75589e9 libc: configurable initial FDs 
The libc already supports the configuration of 'stdin', 'stdout', and
'stderr' using '<libc>' config attributes. This patch equips the libc
with the additional ability to pre-initialize any other file descriptor.
A file descriptor is configured as follows:

<config>
  ...
  <libc ...>
    <fd id="3" path="/dev/log" writeable="yes" readable="no" seek="10"/>
    ...
  </libc>
</config>

Furthermore, this patch moves the FD initialization code from the VFS
plugin to the libc kernel initialization because opening the FDs
depends on 'malloc' ('strdup'), which should not be used at early
'Libc::Kernel' initialization time.

Issue #3478
 2019-08-28 14:18:44 +02:00
Norman Feske
6e38b53001 libc: use Id_space for FD allocator 
This patch replaces the former use of an Allocator_avl with the Id_space
utility, which is safer to use and allows for the iteration of all
elements. The iteration over open file descriptors is needed for
implementing 'fork'.

Issue #3478
 2019-08-28 14:18:44 +02:00
Norman Feske
354e310c87 heap: interface for obtaining backing-store info 
The new 'Heap::for_each_region' method provides information about the
heap's used virtual-memory regions. This method allows for the
mirroring of the heap state as needed by 'fork'.

Issue #3478
 2019-08-28 14:18:44 +02:00
Norman Feske
2cff12e1fb base: remove debug message 2019-08-28 14:18:44 +02:00
Norman Feske
55e99e16ef base: add missing include in base/id_space.h 2019-08-28 14:18:44 +02:00
Norman Feske
e499a04de7 ldso: config option to disarm ctors check 
By specifying the config attribute 'check_ctors="no"', the dynamic
linker won't abort the program on a missing call of
'Env::exec_static_constructors'. This is the case for forked programs
where the ctors were already executed by the forking program prior the
fork operation.

Issue #3478
 2019-08-28 14:18:44 +02:00
Norman Feske
abdf422681 ldso: interface to get infos about loaded objects 
This patch enhances the 'base/shared_object.h' interface of the dynamic
linker with the function 'for_each_loaded_object', which allows the
caller to obtain information about the currently loaded binary and
shared libraries.

The new interface is a base mechanism needed for implementing 'fork' in
the libc.

Issue #3478
 2019-08-21 14:08:18 +02:00
Norman Feske
fd8a209da2 libc: fix warning in socket_fs_plugin.cc 
By using Genode::strncpy instead of the libc's strncpy, we cannot end up
in the situation where the result lacks the zero termination (where the
number of charactors equals the destination buffer size).
 2019-08-21 14:01:54 +02:00
Sebastian Sumpf
15b27a1e9d drivers_nic-imx7d_sabre: increase caps for sel4 2019-08-21 14:00:47 +02:00
Norman Feske
b113f869bb Use I/O signals for resource-request responses 2019-08-21 13:39:44 +02:00
Norman Feske
6fb7022508 Cap-resource request on Pd_session::transfer_quota 
This patch handles the case where a PD's cap quota becomes exhausted
on the attempt to transfer caps via 'Env::pd().transfer_quota'. The
solution mirrors the existing code for RAM quota.

Prior this patch, the 'transfer_quota' operation would trigger an
'Out_of_caps' exception.
 2019-08-21 13:38:24 +02:00
Norman Feske
cc437a5eca Spelling fix in os/single_session_service.h 2019-08-21 13:35:52 +02:00
Christian Helmuth
312f801f8a depot: update recipe hashes 2019-08-21 13:25:26 +02:00
Sebastian Sumpf
8509687c8d system_rtc.run: use packages 
issue #3450
 2019-08-21 13:25:25 +02:00
Martin Stein
ce633c0bba gems: recipes/pkg/drivers_rtc-pc 
Ref #3450
 2019-08-21 13:25:25 +02:00
Sebastian Sumpf
2fc6cedcc0 ld: load dynamic linker at static address on Linux 
* move stack area to static address at beginning of binary
* set ELF type of linker from shared to executable

fixes #3479
 2019-08-21 13:25:25 +02:00
Christian Helmuth
468270f6e9 Enable libusb-internal error login smartcard test 2019-08-21 13:25:25 +02:00
Alexander Boettcher
32323abe8e vfs: handle unsupported watch exception 2019-08-21 13:25:25 +02:00
Martin Stein
1113c4f6a2 depot: mstein download and pubkey 2019-08-21 13:25:25 +02:00
Sebastian Sumpf
e855638266 hw: add system call for irq mode setting 
Core is not allowd to access the kernel's Pic implementation directly.

fixes #3474
 2019-08-21 13:25:25 +02:00
Sebastian Sumpf
f3a7d3750f base: Add Linux and GOT relative startup code on ARM 
* Linux specific startup code for base-linux ARM
* load symbols global offset table relative in order to avoid text
  relocations

issue #3466
 2019-08-21 13:25:25 +02:00
Johannes Kliemann
1bdd18a196 ada-runtime: update to 1.1 2019-08-21 13:25:25 +02:00
Christian Helmuth
0c20bb6ab9 doc: mention BOARD in getting_started.txt 2019-08-21 13:25:25 +02:00
Christian Helmuth
d01cc3bf41 Configurable set_rtc values in RTC test 
Fixes #3450
 2019-08-21 13:25:25 +02:00
Josef Söntgen
9cf5da85ef libc: change time back end to re-read RTC 
Issue #3450
 2019-08-21 13:25:25 +02:00
Josef Söntgen
316f9e4df3 libc: add VFS watch handle accessor 
Issue #3450
 2019-08-21 12:37:03 +02:00
Josef Söntgen
18e586daed libc: merge RTC and time backends 
Issue #3450
 2019-08-21 12:37:03 +02:00
Christian Helmuth
1fdd5b636b vfs: watch set-rtc signal in RTC file system 
Issue #3450
 2019-08-21 12:37:02 +02:00
Christian Helmuth
b8ed80b7dd Simple libc-based RTC test 
Issue #3450
 2019-08-21 12:37:02 +02:00
Martin Stein
cb6377355e sntp_client: report received transmit timestamp 
For now in libports because it relies on contrib sources from musl doing
the time data conversion.

Ref #3448
Ref #3450
 2019-08-21 12:37:02 +02:00
Martin Stein
322bacd380 timeout_types.xsd: Minutes type 
Ref #3448
Ref #3450
 2019-08-21 12:37:02 +02:00
Josef Söntgen
99cb585b6e libports: system RTC server and test 
For now in libports because it relies on contrib sources from musl
doing the time data conversion.

Issue #3450
 2019-08-21 12:37:02 +02:00
Christian Helmuth
0037edfeee libports: minimal time-type conversion from musl 
These are time-type conversion functions (including leap years and
seconds) taken from musl-libc git rev
c82d3bada30cb27e14abda7859da5d2e784830ff with some adaptions. Musl is
licensed under permissive MIT license.

Issue #3450
 2019-08-21 12:37:02 +02:00
Josef Söntgen
a7fe4a502d rtc_drv: implement set signal handling 
Issue #3450
 2019-08-21 12:37:01 +02:00
Josef Söntgen
ea2b330158 rtc: extended session with set-signal 
A client may register a signal handler to be notified whenever the
RTC value was changed, i.e., a mis-configured clock was synchronized,
by calling 'set_sigh()'.

Issue #3450
 2019-08-21 12:37:01 +02:00
Josef Söntgen
86cacd23bb rtc: add documentation comments 
Issue #3450
 2019-08-21 12:37:01 +02:00
Christian Prochaska
85b1563e57 mk: generate .data symbols with distinct values/addresses 
Prevent the static linker from assuming weak object symbols to be alias
symbols for unrelated non-weak symbols with the same value/address.

Fixes #3458
 2019-08-21 12:37:01 +02:00
Christian Prochaska
35c724512d stdcxx: revert '_ZTISt16invalid_argument' symbol type to 'V' 
Issue #3458
 2019-08-21 12:37:01 +02:00
Norman Feske
ff07654560 noux: rm explicit dependency from libc_noux.lib.so 
Issue #3472
 2019-08-21 12:37:01 +02:00
Norman Feske
695a212877 noux: preload libc_noux.lib.so for noux processes 
Issue #3472
 2019-08-21 12:37:01 +02:00
Norman Feske
686dd8affd ldso: ability to preload shared objects 
Fixes #3472
 2019-08-21 12:37:00 +02:00
Christian Prochaska
3acb509b9e vbox: add 'RTMemDupExTag()' implementation 
Fixes #3473
 2019-08-21 12:37:00 +02:00
Norman Feske
874172ca76 sculpt manager: support GEMDOS file systems 
Pass the distinction between GEMDOS and MSDOS file systems from
part-block reports to the vfs_rump plugin.

Fixes #3471
 2019-08-21 12:37:00 +02:00
Norman Feske
114de7721f rump_fs: support the access of GEMDOS file systems 
The GEMDOS variant is supported by NetBSD's msdos file-system driver.
However, it must explicitly be enabled by a mount flag. This patch
adds the principle ability for passing mount flags to file system
drivers and enables the MSDOSFSMNT_GEMDOSFS flag based on the
config attribute 'gemdos="yes"'.

Issue #3471
 2019-08-21 12:37:00 +02:00
Norman Feske
ceae637416 part_block: support for AHDI partition scheme 
This patch enhances part_block with support for parsing the AHDI
partition scheme, and the detection of the GEMDOS variant of FAT as used
by Atari TOS. As a side effect of the implementation, the patch improves
the MBR parsing code by avoiding pointers and using const qualifiers.

Fixes #3470
 2019-08-21 12:37:00 +02:00
Norman Feske
23c2606ce0 usb_block: support iomega zip drives 
This patch enhances the initialization sequence by issueing a START_STOP
SCSI command when needed.

Fixes #3469
 2019-08-21 12:36:59 +02:00
Norman Feske
67a3c2ea4b sculpt: handle failures of usb_block_drv 
This patch handles the situation where the usb_block_drv exits for any
reason, in particular when the driver fails to initialize the device. In
such cases, the usb_block_drv used to stay stale in the system,
effectively preventing the device from being passed to a VM. With the
patch, the USB storage device gets flagged as failed, the usb_block_drv
is removed from the runtime, and the condition is reflected at the user
interface.

This situation occurred on the attempt to access an iomega zip drive
with a version of the usb_block_drv without support for the START-STOP
command, but it may potentially also occur in other circumstances.

Fixes #3468
 2019-08-21 12:36:59 +02:00
Norman Feske
2d03e622f1 usb_block: exit on initialization failure 
Issue #3468
 2019-08-21 12:36:59 +02:00
Christian Prochaska
b9f0318ab8 mk: print symbol assembler file lines only once 
Fixes #3461
 2019-08-21 12:36:59 +02:00
Christian Helmuth
08ac64bba9 libports: download bzip2 port from sourceware.org 
fossies.org removed the archive and Linux From Scratch uses
sourceware.org.

http://www.linuxfromscratch.org/lfs/view/development/chapter03/packages.html
 2019-08-21 12:36:59 +02:00
Stefan Thöni
f018dac506 util/fifo: Change empty() method of fifo to const. 
Fixes #3464
 2019-08-13 12:02:27 +02:00
Christian Helmuth
607fe83c63 autopilot: align output for wide run-script names 2019-08-13 12:02:27 +02:00
Christian Prochaska
f73c63900f abi_symbols: drop weak local symbols 
Fixes #3457
 2019-08-13 12:02:27 +02:00
Christian Prochaska
2fad5eff95 abi_symbols: handle symbols without value 
Fixes #3456
 2019-08-13 12:02:27 +02:00
Christian Prochaska
1f56ffa51a abi_symbols: better error message on failed regexp match 
Fixes #3455
 2019-08-13 12:02:27 +02:00
Christian Helmuth
ce149397ec acpica: retrieve static battery info only once 
On Fujitsu S936, evaluating the _BIF method on each battery-info update
successively increased the RAM usage, while  _BST (dynamic battery
status) and _STA (generic status) did not show this behavior. Therefore
this commit retrieves only dynamic information periodically (resp. on
SCI IRQ). Now, acpica hast static RAM usage in idle state for 24+ hours.

The root cause for the increased RAM usage is still shady. While it
could just be normal that it grows until a certain yet unknown limit,
there may also be memory leak in contrib code or some strange AML on the
designated notebook.

Issue #3454
 2019-08-13 12:02:27 +02:00
Christian Helmuth
a7835650e8 acpica: prevent contrib code warnings 2019-08-13 12:02:27 +02:00
Christian Helmuth
83ead086a1 acpica: enable interpreter-slack mode 
This mode is used on Linux (if acpi=strict is not set on boot cmdline)
and Windows. The mode ignores certain errors and/or bad AML constructs.

  1) Allow "implicit return" of last value in a control method
  2) Allow access beyond the end of an operation region
  3) Allow access to uninitialized locals/args (auto-init to integer 0)
  4) Allow ANY object type to be a source operand for the Store() operator
  5) Allow unresolved references (invalid target name) in package objects
  6) Enable warning messages for behavior that is not ACPI spec compliant
 2019-08-13 12:02:27 +02:00
Christian Prochaska
91c8e70bef depot: add recipe for drivers_nic-imx8q_evk 
Fixes #3452
 2019-08-13 12:02:27 +02:00
Christian Prochaska
38dcdeeb04 fec_nic_drv: arm_64 support 
Fixes #3446
 2019-08-13 12:02:27 +02:00
Christian Helmuth
1c77ea2b03 base-hw: remove other board libs from recipes 2019-08-13 12:02:27 +02:00
Christian Helmuth
82d50912f6 foc: exclude .git directories from src archives 2019-08-13 12:02:27 +02:00
Christian Prochaska
4c113182b0 depot: add recipe for base-hw-imx8q_evk 
Issue #3426
 2019-08-13 12:02:27 +02:00
Stefan Kalkowski
7ced122ddc hw: support for i.MX8M Quad EVK 
Fix #3426
 2019-08-13 12:02:27 +02:00
Stefan Kalkowski
6b09ac59f0 hw: enable performance counter for ARMv8 
Ref #3426
 2019-08-13 12:02:27 +02:00
Stefan Kalkowski
ee38504d81 hw: implement update_data_region for ARMv8 
Ref #3426
 2019-08-13 12:02:26 +02:00
Sebastian Sumpf
dd505edd19 hw: GICv3 implementation 
* modern GICv3 implementation
* distributor
* redistributor
* MMIO cpu interface

Ref #3426
 2019-08-13 12:02:26 +02:00
Stefan Kalkowski
fa1aa33f83 hw: sanitize arm trustzone/virtualization services 
Those services are not SoC specific and have to reside at a generic place.

Fix #3445
 2019-08-13 12:02:26 +02:00
Stefan Kalkowski
907de9d37f hw: move timer into board.h 
Unify the generic timer implementation for ARMv7 and ARMv8.

Ref #3445
 2019-08-13 12:02:26 +02:00
Stefan Kalkowski
5c7436bf10 hw: remove SMP variable from board.h 
Whether an SoC has the multiprocessing extensions can be read out
from the identification registers, and does not need to be specified
in each board header.

Ref #3445
 2019-08-13 12:02:26 +02:00
Stefan Kalkowski
0b77e8ea62 hw: consistently move cpu into board namespace 
Ref #3445
 2019-08-13 12:02:26 +02:00
Stefan Kalkowski
875858b2cc hw: integrate interrupt controllers into board.h 
Additionally, unify more implementation details in between different
usage patterns of ARM's generic interrupt controller (v2)

Ref #3445
 2019-08-13 12:02:26 +02:00
Martin Stein
fe426e6f8f smbios_decoder: XML report from SMBIOS table ROM 
Fixes #3430
 2019-08-13 12:02:26 +02:00
Martin Stein
1e379cb3a9 drivers/acpi: provide plain SMBIOS table as report 
Ref #3430
 2019-08-13 12:02:03 +02:00
Martin Stein
ead385dd17 base-nova: EFI sys-table pointer in platform info 
Ref #3430
 2019-08-13 12:02:03 +02:00
Martin Stein
b87e21a392 base-hw: EFI sys-table pointer in platform info 
Ref #3430
 2019-08-13 12:02:03 +02:00
Christian Prochaska
04e8ba716c ps2_drv: wait longer for ACK after RESET command 
On some machines, more time passes until the acknowledgement arrives.

Fixes #3442
 2019-08-13 12:02:03 +02:00
Christian Prochaska
193a401097 Increase RAM quota of ACPI driver to 4M 
On some machines, more than 3 MiB are needed. This finally equalizes all
occurences of RAM quotas with drivers_managed.

Fixes #3443
 2019-08-13 12:02:03 +02:00
Alexander Boettcher
53a83fb76e nova: update kernel branch 
- Avoid DoS against kernel, reported by Julian Stecklina
- Provide EFI systab pointer through HIP
- Fix PCID bug

Fixes #3440
 2019-08-13 12:02:03 +02:00
Josef Söntgen
47e6d72bf2 rtc_drv: add support for setting RTC 
The rtc_drv on x86 can now by used to also set the RTC. If the config
attribute 'allow_setting_rtc' is set to 'yes' the driver will update
the RTC from the content of the 'set-rtc' ROM module. A valid ROM must
contain a top node with the following attributes: 'year', 'month',
'day', 'hour', 'minute' and 'second'.

* Only rudimentary checking of the provided values is done.
* '12H' mode is not supported.

Fixes #3438.
 2019-08-13 12:02:03 +02:00
Josef Söntgen
91ce57848c rtc_session: make timestamp printable 
Issue #3438.
 2019-08-13 12:02:03 +02:00
Martin Stein
22d7871e1d test/init: raise log performance 
Fixes #3429
 2019-08-13 12:02:03 +02:00
Tomasz Gajewski
4189157d10 cpu_bench: fixes to make it work reliably 
Added missing build dependency.

Eliminated race condition when processing test output.

Fixes #3412
 2019-08-13 12:02:03 +02:00
Sid Hussmann
eaefcc2c6f ssh: add exec channel and exit on interactive 
This commit implements the ssh exec channel request. It also handles
some shortcommings on the interactive channel like exit and concurrent
session establishments.

Pipes into the channel do not work yet. E.g.:
echo foobar | ssh noux@localhost -p 5555 "cat > /rw/test.txt"

The issue described with FIXME in Ssh::Server::incoming_connection()
could not be reproduced and might have been fixed with the improved
file descriptor handling.

Fixes #3401
 2019-08-13 12:02:03 +02:00
Christian Helmuth
92bdcbf1fe ps2: initially set all keyboard LEDs off 
This ensures a safe initial state and also informs the BIOS (resp.
notebook keyboards with dual-use keys for numpad) about our
assumptions.
 2019-07-09 15:57:32 +02:00
Norman Feske
c011d54158 sculpt/README: refer to cnuke's updated article 2019-07-09 14:18:31 +02:00
Norman Feske
d782499541 sculpt/README: specify BOARD for building 2019-07-09 13:59:59 +02:00
Norman Feske
59f9af23b9 News item for Sculpt 19.07 2019-07-09 12:20:06 +02:00
Norman Feske
70a236cccd sculpt/README: adjust git tag to version 19.07 2019-07-09 11:59:44 +02:00
Norman Feske
d33bef2e49 Update documentation for Sculpt 19.07 2019-07-09 11:46:20 +02:00
Christian Helmuth
fbae7767b4 sculpt: version 19.07 2019-07-09 09:06:54 +02:00
Christian Helmuth
17d32b3e15 depot: update recipe hashes 2019-07-09 09:06:54 +02:00
Norman Feske
6380ad1c80 pkg/vbox5-nova-sculpt: clipboard support 
To enable the clipboard for a VM, add the following node to the
<Hardware> sub node of your machine.vbox configuration:

  <Clipboard mode="Bidirectional"/>

Issue #3437
 2019-07-09 08:58:39 +02:00
Norman Feske
0c3d3e7c8f pkg/qt5_textedit: clipboard support 
Issue #3437
 2019-07-09 08:58:39 +02:00
Norman Feske
3b0d694f42 pkg/noux-system: clipboard support 
Issue #3437
 2019-07-09 08:58:39 +02:00
Norman Feske
c7e79030dd sculpt: limit rate of depot queries 
The triggering of a new depot query can happen more than once per
activation of the sculpt manager if multiple conditions call for updated
information about the depot. When this happens, the depot-query
component produces intermediate results, which are not consumed by the
sculpt manager. By deferring depot queries for a few milliseconds, we
avoid such intermediate queries, relieving the workload of the
depot-query component at system boot time.

Issue #3436
 2019-07-09 08:58:39 +02:00
Norman Feske
58a0f5c30b sculpt: break possible depot-query feedback loop 
Fixes #3436
 2019-07-09 08:58:39 +02:00
Norman Feske
c92a9ce591 sculpt: show names of unsatisfied launchers 
The diagnostic messages presented in the runtime dialog lacked the name
if the subsystem was created from a launcher, e.g., the 'vm'. Instead of
determining the subsystem name from the start-XML-node (a launcher has
no 'name' attribute but the name corresponds to the launcher's file
name), the name is now passed as a dedicated argument.
 2019-07-09 08:58:39 +02:00
Norman Feske
605f9abf96 sculpt: integrate global clipboard 
Issue #3437
 2019-07-09 08:58:39 +02:00
Norman Feske
e4412a3e56 noux_vim.run: clipboard integration with wm 
This patch enhances the noux_vim.run scenario to make it usable as a
testing ground of the terminal's and window manager's resize and
clipboard handling.

Issue #2079
 2019-07-09 08:58:39 +02:00
Norman Feske
1fe9b07c99 wm: forwarding of clipboard ROM/Report sessions 
The labels of clipboard ROM and clipboard report sessions of WM clients
must be consistent with the client's nitpicker label. Hence, we must
route those sessions through the window manager, analogously to the
approach taken for shape reports in #3165.

Issue #3437
 2019-07-09 08:58:38 +02:00
Norman Feske
2da604a6e2 terminal: make color 4 less green, less saturated 
Issue #3406
 2019-07-09 08:58:38 +02:00
Norman Feske
fc7b983a40 terminal: clipboard support 
Fixes #2079
 2019-07-09 08:58:38 +02:00
Norman Feske
6399fc12ac clipboard: allow for dynamic reconfiguration 2019-07-09 08:58:38 +02:00
Norman Feske
db6e013577 clipboard: match client labels against focus 
When setting the config attribute 'match_labels="yes"', the clipboard
performs plausibility checks for copy and paste operations against the
nitpicker focus.

Issue #3437
 2019-07-09 08:58:38 +02:00
Christian Prochaska
71ce1442c8 run: enable 'imx7d_sabre' in network tests 
Fixes #3434
 2019-07-09 08:58:38 +02:00
Christian Prochaska
5dd1abcc2d depot: add recipe for drivers_nic-imx7d_sabre 
Fixes #3433
 2019-07-09 08:58:38 +02:00
Christian Prochaska
4dafdbd5b7 dde_linux: update 'fec' driver to version 4.16.3 
Fixes #3432
 2019-07-09 08:58:38 +02:00
Christian Helmuth
949130d80e Remove warning exec_static_constructors() warning 
The warning falsely detected cases where shared objects where loaded before
exec_static_constructors() was called as unneeded even in cases were the
binary itself contained static globals.

The commit also removes one redundant call to exec_static_constructors()
from the block tester.
 2019-07-09 08:58:38 +02:00
Josef Söntgen
4c8369ab1b tool/depot/publish: support multiple threads in xz 
Introduce 'XZ_THREADS' to override the default number of threads used
for compression, which is still set to '1'. As using multiple threads
will increase the amount of memory needed during the compression and
potentially influences the size of the created archive, setting the
variable limits the impact.

Fixes #3431.
 2019-07-09 08:58:38 +02:00
Christian Helmuth
fcdad00044 test.run: build vfs plugins of in rom list 2019-07-09 08:58:38 +02:00
Norman Feske
417191bdbd wm.run: build window_layouter 
This change makes the wm.run more convenient as testbed for developing
the window layouter.
 2019-07-09 08:58:38 +02:00
Norman Feske
c00666b7e8 depot_deploy.run: integrate recall_fs 
This is needed to deploy the current version of the window_layouter pkg.
 2019-07-09 08:58:38 +02:00
Norman Feske
854a154fb4 depot_query: cache file-system accesses 
This patch introduces two caches to the depot-query tool.

- A stat cache remembers the results of 'Directory::file_exists'
  calls.

- The 'Cached_rom_query' caches the result of scanning the depot
  for a given ROM module and pkg path. To elminates the need to
  parse 'archive' files of pkgs referenced from other pkgs or
  for the repeated instantation of the same pkg.

Both caches are bypassed whenever referring to the 'local' depot user.

Fixes #3427
 2019-07-09 08:58:38 +02:00
Norman Feske
7e7eff0eb7 window layouter: reduce snap-back artifact 
When resizing windows of clients that respond very slowly to resize
requests, the window's size sometimes snapped back to its original size
immediately after finishing the drag operation.

The problem was caused by the interplay of the layout rules (obtained
via the 'rules' ROM, generated by the 'rules' report) and the
temporary interactive state that occurs during drag operations.
The rules are updated only at the time of releasing the button to keep
the overhead while dragging the window low. However, when releasing the
mouse, the (now outdated) rules kicked back into effect, triggering
resize requests for the window to its old size.

The patch solves this problem by decoupling the dragged state of a
window from the physical release of the button. The button release
triggers a transition from a DRAGGING to a SETTLING state and programs
a timer. In the SETTLING state, the windows behave as in DRAGGING state,
giving the interactive geometry precedence over the rules-dictated
geometry. During this state, further responses of window-resize requests
may come in and are handled like dragging was still in progress. After a
timeout, however, the current window layout is conserved as a new rules
report and the state goes back to IDLE.

For clients that takes a very long time (in particular, VirtualBox when
resizing the desktop, which takes sometimes multiple seconds), the
snap-back artifact can still occur, but the effect is reduced.
 2019-07-09 08:58:38 +02:00
Norman Feske
48a361107f gpt_write: fix false-positive fallthrough warning 2019-07-09 08:58:38 +02:00
Norman Feske
716453aaa1 gpt_write: fix pointer to local variable 
This problem surfaced with the new tool chain that changes the stack
layout. A pointer to the the config XML data was kept in the main object
but pointed to a stack variable. This patch fixes it by removing the
pointer.

Fixes #3416
 2019-07-09 08:58:38 +02:00
Christian Helmuth
63e69d35e0 Disable unsupported autopilot tests on rpi3 2019-07-09 08:58:38 +02:00
Christian Helmuth
a0c71cffd4 Solo5 tests require x86_64 2019-07-09 08:58:38 +02:00
Norman Feske
d1da5b9e16 libc_noux: override __libc_sigaction 
This is a follow-up commit to the libc update. The FreeBSD
implementation of sigset or sigaction calls the __libc_sigaction now.

The patch fixes the SIGWINCH handling for Vim when interactively
resizing the terminal window.

Fixes #3424
 2019-07-09 08:58:38 +02:00
Norman Feske
52ec4526e1 libc: discharge use of hidden symbols 
Issue #3424
 2019-07-09 08:58:38 +02:00
Martin Stein
e199acca05 util/print_lines: don't skip end of long lines 
Print the end of oversized lines (> 200 chars) to new lines instead of
skipping it

Fixes #3422
 2019-07-09 08:55:24 +02:00
Christian Helmuth
cea842b256 autopilot: append RUN_OPT also with --keep 
In case RUN_OPT_AUTOPILOT is changed on consecutive calls of "autopilot
--keep", the changes are now effectively propagated to the run tool.

I also documented the RUN_OPT_AUTOPILOT environment variable.
 2019-07-09 08:55:24 +02:00
Josef Söntgen
984bddcedd libc: cleanup port preparation 
- make port able to be prepared repeatedly
  Rpcgen refuses to overwrite an already existing header, so try to
  always remove it beforehand
- get rid of 'cp' warning 'cp: warning: source file
  'src/lib/libc/sys/sys/time.h' specified more than once'
- silence patch message 'patch unexpectedly ends in middle of line'

Fixes #3420.
 2019-07-09 08:55:24 +02:00
Josef Söntgen
d32e8390c2 os: remove trace_fs 
This component is superseded by the 'vfs_trace' plugin.

Issue #3294.
 2019-07-09 08:55:24 +02:00
Sebastian Sumpf
57e650ef2c gems: recipes for vfs_trace pluging 
* source recipe for vfs_trace
* pkg for 'trace_fs' with runtime

issue #3294
 2019-07-09 08:55:24 +02:00
Sebastian Sumpf
c34176580c os: recipe for 'trace_policy' 
* source packet for all trace policies

issue #3294
 2019-07-09 08:55:24 +02:00
Emery Hemingway
807f5d9af1 Use pthread_join in test/libc_pipe 
Ref #2303
 2019-07-09 08:55:24 +02:00
Norman Feske
d18f8eea2b window layouter: control maximize by editing rules 
This patch adds the missing propagation of the maximized state from the
layout rules to the internal representation of a window. Without this
patch this state could be toggled by clicking on the maximizer button
only.
 2019-07-09 08:55:24 +02:00
Christian Helmuth
8d62f21b40 autopilot: support arch-specific build directories 
Replace the notion of board-specific platforms by clear definition
architecture and board.

Also added new command line switch for "targets"

  -t <target>         test target as triple of architecture-board-kernel,
                      e.g., arm_v7a-pbxa9-hw

Related to #3316
Fixes #3417
 2019-07-09 08:55:24 +02:00
Norman Feske
7b1e3a9d63 window layouter: resize resizeable windows only 
Initiate the drag operation of a resize handle only if the client is
known to be resizable.

Issue #3303
 2019-07-09 08:55:24 +02:00
Norman Feske
070a24956d themed_decorator: use Attached_rom_dataspace::xml 
This simplifies the code a bit.
 2019-07-09 08:55:24 +02:00
Norman Feske
dec2d897c3 themed_decorator: fix jumping windows 
The decorator's detection of the hovered window element was inaccurate,
which resulted in "jumping" windows in some situations, ultimately
caused by a combination of three different mechanisms.

First, when moving the pointer into the area of a window, the decorator
would detect the hovering of the left border whenever the distance of
the pointer from the border was less than the half of the theme texture
(e.g., 64x64 pixels for the default theme). However, if the left border
margin is set to a small value (e.g., 1), there is an overlap of the
sensitive resize border area and the content. Hence, chances were quite
high that - when moving the pointer from the left into the window - the
hover report would contain the hovering of the left border.

Second, the window manager tries to hide pointer movements from the
decorator if possible. It informs the decorator of the pointer position
if any decoration is hovered or if a new window is hovered. But it does
not expose pointer movements within a window to the decorator. For this
reason, the decorator would not update the hover report as long as the
pointer stays within a once hovered window. In the situation described
above, the hover report would still contain the stale information about
the hovering of the left resize border.

Third, when the user clicks on the window, the decorator examines the
most recent hover report and - in the situation described above - finds
the left border hovered. Consequently, it initiates a window-drag
operation. While resizing the window with the left border, the window
layouter pins the right border of the window to its current position.
All window-size changes of the client will be applied towards the left
(dragged) border. In the case of the top view, which continuously
resizes the window by itself, the window would "jump". In reality, it
actually tries to respond an interactive resize operation. The window
layouter cannot guess that the client is not responding to window
layouter's resize request but is acting independently.

This patch fixes the jumping window problem for the case where the
pointer hovers the overlapping area of the resize border and the
content. However, when trying the to interactively resize the top window
via the bottom-left corner, the "jumping" can still be observed.

Fixes #3303
 2019-07-09 08:55:24 +02:00
Norman Feske
8625146011 sculpt: use secondary CPU for leitzentrale 
By decoupling the leitzentrale from the (contended) boot CPU, the fading
on F12 interferes much less with animations like nano3d deployed in the
runtime.

Fixes #3268
 2019-07-09 08:55:24 +02:00
Sebastian Sumpf
24eea0b653 base: add 'trace' function to base/log.h 
The 'Genode::trace' convenience function prints messages to Genode's
trace buffer (if tracing is enabled).

issue #3294
 2019-07-09 08:55:23 +02:00
Sebastian Sumpf
18b3253cac vfs_trace: VFS plugin that offers trace buffer access 
The plugin creates a file-system hierarchy that enabled the access of
trace buffers for each component and its threads.

issue #3294
 2019-07-09 08:55:23 +02:00
Sebastian Sumpf
25484f870e trace: make trace buffer resizeable 
Trace buffers a re-allocated during subsequent calls to 'trace'.

issue #3294
 2019-07-09 08:55:23 +02:00
Sebastian Sumpf
ac0ecdf855 base: make trace resumable 
revert relocations so a paused subject can resume without changing its
policy.

issue #3294
 2019-07-09 08:55:23 +02:00
Sebastian Sumpf
399487c85d gems: add direct fs constructor to Watch_handler 
issue #3294
 2019-07-09 08:55:23 +02:00
Alexander Boettcher
bc14a32998 bomb: handle failing child creation 
Issue #3107
 2019-07-09 08:55:23 +02:00
Christian Helmuth
cf2c9f8d88 run: abort with error if BOARD was not set 2019-07-09 08:55:23 +02:00
Stefan Kalkowski
186c35bb09 depot: recipe for base-hw-rpi3 
Ref #3407
 2019-07-09 08:55:23 +02:00
Sebastian Sumpf
3a2de0d9a3 depot_autopilot: limit dynamic to 286 MB 
* also set test-libc to 256 MB (seems to be enough)
* fixes memory issues on platforms with 512 MB RAM

issue #3407
 2019-07-09 08:55:23 +02:00
Sebastian Sumpf
1b489820a9 base: timestamp support for ARM 64-Bit 
issue #3407
 2019-07-09 08:55:23 +02:00
Sebastian Sumpf
ef8140cce5 lxip: ARM 64-Bit support 
issue #3407
 2019-07-09 08:55:23 +02:00
Sebastian Sumpf
790a57041c base: add ARM 64-Bit to base API 
issue #3407
 2019-07-09 08:55:23 +02:00
Sebastian Sumpf
e43b848ac8 jitterentropy: ARM 64-Bit version 
issue #3407
 2019-07-09 08:55:23 +02:00
Sebastian Sumpf
59347d2560 stdcxx: make linkable for ARM 64-Bit 
'typeinfo for std::invalid_argument' was not found when using 'V' (weak
object)

issue #3407
 2019-07-09 08:55:23 +02:00
Sebastian Sumpf
2b14284e2b libc: add ARM-64 bit to recipe 
issue #3407
 2019-07-09 08:55:23 +02:00
Sebastian Sumpf
c107aba913 depot_autopilot: add ARM 64-bit support 
issue #3407
 2019-07-09 08:55:22 +02:00
Sebastian Sumpf
e337b844e5 depot: support for ARM 64-bit for base-foc on rpi3 
issue #3407
 2019-07-09 08:55:22 +02:00
Stefan Kalkowski
90d07741aa hw: support for ARM64 Raspberry Pi 3 
Restriction: enables only cpu core 0 and the timer interrupt by now.

Fix #3405
 2019-07-09 08:55:22 +02:00
Stefan Kalkowski
87015df66c hw: change update_pd to invalidate_tlb 
In the past, the core-only privileged syscall `update_pd` was used only
to invalidate the TLB after removal of page-table entries.
By now, the whole TLB at least for one protection domain got invalidated,
but in preparation for optimization and upcomingARM v8 support,
it is necessary to deliver the virtual memory region that needs to get
invalidated. Moreover, the name of the call shall represent explicitely
that it is used to invalidate the TLB.

Ref #3405
 2019-07-09 08:55:22 +02:00
Stefan Kalkowski
d9a0f76e7a hw: extend long-descriptor page table format 
Ref #3405
 2019-07-09 08:55:22 +02:00
Tomasz Gajewski
0ecc48e6de Mini UART driver for Rpi3 
Ref #3405
 2019-07-09 08:55:22 +02:00
Christian Helmuth
efe0887247 tool: fix BOARD_RUN_OPT for arm_v8 2019-07-09 08:41:53 +02:00
Christian Helmuth
3c4c460f82 depot: update recipe hashes 2019-06-13 13:40:37 +02:00
Christian Helmuth
ab3efc0a33 depot_autopilot: skip test-fs_packet on rpi 2019-06-13 12:22:50 +02:00
Christian Helmuth
e1e67d8330 Increase timeouts in test-libc* pkgs 
These tests often fail with Qemu on aged hosts because of the tight
timeouts.
 2019-06-13 12:22:50 +02:00
Christian Helmuth
de4c31e557 Additional boot time (10 s) in test.run 2019-06-13 12:22:50 +02:00
Martin Stein
5a59d295a4 depot_autopilot.run: relax test timeout 
There are still nightly tests like test-tcp_bulk_lxip on sel4 x86_64 qemu
that manage to hit the test timeout of the run script although the test was
successful. So, raise the extra time added by the run script to 30 seconds.

Ref #3411
 2019-06-13 12:22:49 +02:00
Martin Stein
fe5ada85cb depot_autopilot.run: max nr of tests per boot 
Some platforms (sel4 imx6/imx7) cannot manage to execute all tests in a single
boot. Thus, we re-boot them periodically after a given maximum number of tests
to avoid that arbitrary tests always fail due to the long uptime and not due to
the tests themselves.

If the maximum number of tests is set to 0, no limit is applied.

Fixes #3411
 2019-06-13 12:22:49 +02:00
Johannes Kliemann
b55358c535 ada-runtime: fix exception handling on ARM 
ref #3395
 2019-06-13 12:22:49 +02:00
Christian Helmuth
1b059649ff depot_autopilot: skip test-spark_exception on ARM 
Issue #3395
 2019-06-13 12:22:49 +02:00
Christian Helmuth
45f3ccc42a vfs: fix empty-node handling in inline fs 
Now <inline></inline> as well as <inline/> are supported.
 2019-06-13 12:22:49 +02:00
Christian Helmuth
4d34f9f9f4 Reduce third-party kernel build warnings 2019-06-13 12:22:49 +02:00
Christian Prochaska
62232f9a35 fs_rom: retry watching a path element after watching its parent dir 
If a path element cannot be watched because it does not exist yet, the next
available parent directory is watched to detect the creation of the missing
path element. If a missing path element got created just between the failed
watch request for the element and the successful watch request for its parent
directory, the creation of the path element could remain unnoticed.

With this commit, 'fs_rom' tries to watch a path element again if the watch
request for its parent directory was successful.

Fixes #3398
 2019-06-13 12:22:49 +02:00
Alexander Boettcher
a25c656b1a image/disk: increase estimated disk size 
Fixes #3410
 2019-06-13 12:22:49 +02:00
Christian Helmuth
ed0d76552b Relax timing requirements in lazy timer test 
The lazy-timer test depends on the faster-timer handler to be executed
before the fast timeout occurs, which was pretty hard to achieve on Qemu
and a busy host machine. Therefore, I increased the fast-to-faster
timeout ratio from 50/25 ms to 200/25 ms and set the test runtime to
4000 ms.
 2019-06-13 12:22:49 +02:00
Emery Hemingway
43315444a8 Libc: wrap VFS sync's in VFS_THREAD_SAFE 
Move the _vfs_sync implementation from vfs_plugin.h to vfs_plugin.cc and
wrap all VFS access in the VFS_THREAD_SAFE macro.

Syncing must be made thread safe because it often happens when closing
files and may require suspending the current thread for I/O signal
dispatching.

Ref #3409
 2019-06-13 12:22:49 +02:00
Alexander Boettcher
c055986e38 vbox5*.run: enable VM tests for Fiasco.OC 
Issue #3111
 2019-06-13 12:22:49 +02:00
Alexander Boettcher
7ec37b2d8d vbox5: adjust to run with Fiasco.OC 
- use priorities solely with nova
- handle exceptions during vCPU creation
- avoid assertion in vga_vbda
- silence compiler warnings about outdated register keyword

Issue #3111
 2019-06-13 12:22:49 +02:00
Alexander Boettcher
64fac6cee7 foc: extend vm_session impl. for vbox5 
- transfer more guest registers
- print warning once for unsupported guest registers
- improve synchronization of vm state transfer

Issue #3111
 2019-06-13 12:22:49 +02:00
Emery Hemingway
12127a7bd2 Construct Terminal_session within with_libc 
The "pipe" procedure can only be safely called within with_libc.

Fix #3408
 2019-06-13 12:22:49 +02:00
Josef Söntgen
eaed28f68d sculpt: document version string constraint 
Issue #3403.
 2019-06-13 12:22:49 +02:00
Tomasz Gajewski
4a102fb58c sculpt doc: be more precise about the use of GPT 
Information about GPT being required for permanent sculpt
customizations

Fixes #3381
 2019-06-13 12:22:48 +02:00
Christian Helmuth
ae7fbd6b1a terminal: modernize color palette 
The terminal now got a configurable palette for 16 colors (8 normal, 8
bright/bold).

  <config>
    <palette>
      <color index="0" value="#000000"/> <!-- black is real black -->
      <color index="8" value="#101010"/> <!-- bright black stands out a bit -->
    </palette>
  </config>

Note, the old (undocumented) <color index="..." bg="..."> configuration
scheme is no longer supported.

Also, this commit adds a pleasing default palette that ensures
readability of ViM's standard hightlighting.

Fixes #3406
 2019-06-13 12:22:48 +02:00
Christian Helmuth
72478a9e31 nova: provide missing Rpc_entrypoint::is_myself() 
The symbol is implemented by the common implementation but was missing
from the nova-specific fork of rpc_entrypoint.cc.
 2019-06-13 12:22:48 +02:00
Stefan Kalkowski
811410fdbb foc: update to current github version (fix #3402) 
Instead of hosting our outdated own fork, this commit now uses kernkonzept's
nowadays available github source repositories. This hopefully simplifies
updates, or cherry-picking single fixes.
 2019-06-13 12:22:48 +02:00
Christian Helmuth
5738e53be2 Warn about questionable call of wait_and_dispatch_one_io_signal 
Issue #2399
 2019-06-13 12:17:32 +02:00
Christian Helmuth
be4d8b24e2 run: improve error message on missing recipe 2019-06-13 12:17:32 +02:00
Christian Helmuth
1f77f37310 qt5: handle keypad-enter key 2019-06-13 12:17:32 +02:00
Norman Feske
7e174e73be ttf_font: prevent out-of-bounds access 
Thanks Alexander Böttcher for investigating.

Fixes #3393
 2019-06-13 12:17:31 +02:00
Emery Hemingway
ff23d665c6 depot: move ehmry depot URL 
Fix #3400
 2019-06-13 12:17:31 +02:00
Christian Helmuth
00daecf838 qt5: enable capslock/numlock in test run scripts 2019-06-13 12:17:31 +02:00
Christian Helmuth
f8f9f710e5 vbox5: rewrite (truncate) nitpicker session label 
The shape-report label matching depends on the truncated label, i.e.,
Nitpicker label "wm -> vm" matches, while "wm -> vm -> vbox" does not.
 2019-06-13 12:17:31 +02:00
Alexander Boettcher
e77362e7b9 driver_manager: fix vesa_fb_drv binary usage 
Fixes #3394
 2019-06-13 12:17:31 +02:00
Alexander Boettcher
c58acbd444 sculpt: add VM service 
Fixes #3396
 2019-06-13 12:17:31 +02:00
Alexander Boettcher
e2c1ce2add foc: add core threads as trace source 
Fixes #3390
 2019-06-13 12:17:31 +02:00
Alexander Boettcher
fc1b7a626c foc: add idle times as trace source 
Issue #3390
 2019-06-13 12:17:31 +02:00
Christian Prochaska
a7825ef292 gdb_monitor: disable pagination earlier 
Fixes #3397
 2019-06-13 12:17:31 +02:00
Christian Prochaska
8dd391371c libusb: show "device is plugged" message only when plugged 
Fixes #3399
 2019-06-13 12:17:31 +02:00
Josef Söntgen
737e9580b8 nvme_drv: only set the first INTMS bit on reset 
Since we only set the first INTMC bit a check in QEMU will fail
and we don't get interrupts.

Fixes #3369.
 2019-06-13 12:17:31 +02:00
Josef Söntgen
a2abc13425 nvme_drv: only wait for first admin cmd 
Leave the loop if we processed the cmd we were looking for. That
should fix the delay users notice in Sculpt when NVMe is used.

Issue #3369.
 2019-06-13 12:17:31 +02:00
Josef Söntgen
1daf179db9 nvme_drv: adapt run script to block_tester config 
Issue #3369.
 2019-06-13 12:17:31 +02:00
Josef Söntgen
6b48a08697 libc: silence noncritical dummy implementations 
* isatty

  For the moment it is not possible to determine if the fd belongs to a
  tty. For whatever reasons, the check is done multiple times, e.g.
  'tclsh', which will spam the LOG.

* sysctl(HW_FLOATINGPT)

  FPU is enabled on all our platforms, so return true.

Fixes #3389.
 2019-06-13 12:17:31 +02:00
Emery Hemingway
dc8a2ec523 Remove libc_terminal 
Fix #3378
 2019-06-13 12:13:40 +02:00
Christian Prochaska
46f89d143b libports: ncurses: disable relative cursor movement 
Fixes #3380
 2019-06-13 12:13:40 +02:00
Emery Hemingway
8a4adceaf0 Add note on PCAP dump to run/fetchurl.inc 
Ref #3388
 2019-06-13 12:13:40 +02:00
Emery Hemingway
bbf7a6230e VFS lwIP: restart DHCP with link-state changes 
Start and stop the DHCP state machine as the Nic link-state changes.
Invoke the link state handler during configuration rather than assume
that in the case of a downed link lwIP will defer DHCP until the link
comes up.

Additionally, support static DNS configuration via the "nameserver"
configuration attribute.

Fix #3388
 2019-06-13 12:13:39 +02:00
Christian Helmuth
24a3eea89e version: 19.05 2019-05-29 11:03:29 +02:00
Norman Feske
65b9a46d6c News item for version 19.05 2019-05-29 11:02:24 +02:00
Norman Feske
4dbb18bf5e Release notes for version 19.05 2019-05-29 11:02:24 +02:00
Norman Feske
44181fedf7 doc: update tool_chain.txt 
This commit updates the tool-chain information, pointing at the
versioned install path now.

It also removes historic information that is no longer of interest.
 2019-05-29 11:02:24 +02:00
Christian Helmuth
2b183f9497 depot: update recipe hashes 2019-05-29 10:20:52 +02:00
Martin Stein
a09150e6b4 depot_autopilot.run: relax reboot timeout 
It might happen, in CPU intensive tests (like TCP bulk lxip), or when
printing debugging output after a failed test (as done currently on
staging), that the run script on the host gives up and reboots the
platform too early. Thus, we raise the buffer time. A reboot should
be necessary only in rare cases anyway.

Fixes #3387
 2019-05-29 10:20:52 +02:00
Martin Stein
da3c52668d depot_autopilot.run: fix insufficient Qemu RAM 
Fixes an error output in init at the beginning of the test and insufficient
transfer of RAM to the runtime of RAM-intensive test packages.

Ref #3387
 2019-05-29 10:20:52 +02:00
Martin Stein
5ba19dd49b test-nic_loopback: fix RAM resource request 
Ref #3387
 2019-05-29 10:20:52 +02:00
Sebastian Sumpf
4b72bbaa57 libc: mktime signed overflow 
issue #3289
 2019-05-29 10:20:52 +02:00
Alexander Boettcher
9f02151d0b seoul-auto: disable autopilot test on qemu x86_32 
Issue #3373
 2019-05-29 10:20:52 +02:00
Sebastian Sumpf
f75f477c2b nic_router: [[fallthrough]] annotation 
issue #3377
 2019-05-29 10:20:52 +02:00
Sebastian Sumpf
cec050983a hw: [[fallthrough]] annotations for ARM/RISC-V 
* double checked

issue #3377
 2019-05-29 10:20:52 +02:00
Christian Prochaska
ca39a9ea61 tool_chain: versioned install location 
Issue #3307
 2019-05-29 10:20:52 +02:00
Martin Stein
ca95621db5 run/nic_router_uplinks: do not run on muen 
On muen, we don't have configured access to wifi so far. Thus, the wifi
driver doesn't come up and we should skip tests using this driver for
the platform.

Fix #3384
 2019-05-29 10:20:52 +02:00
Tomasz Gajewski
27e9d1fcf0 tool/run/README: Fixed tftp related argument names in example 
Fixes #3382
 2019-05-29 10:20:52 +02:00
Christian Prochaska
f99a1d78bb os: fix 'CUU' terminal command 
Decrement the y position to move the cursor up.

Fixes #3379
 2019-05-29 10:20:52 +02:00
Sebastian Sumpf
d417d26ce8 hw: fix calculation of CPU count on x86_64 
On x86 the CPU count is determined through ACPI's MADT by counting the
local APICs reported there. Some platforms report more APICs
than there are actual CPUs. These might be physically disabled CPUs.
Therefore, a check if the LAPIC is actually physically enabled in
hardware fixes this issue.

Thanks to Alex Boettcher

fixes #3376
 2019-05-29 10:20:52 +02:00
Sebastian Sumpf
5611020f33 hw: fix stack alignment in bootstrap for x86_64 
Fix initial stack pointer alignment for x86_64 in crt0.s startup code of
bootstrap. SysV ABI states that upon function entry (rsp + 8) % 16 = 0.
There, we have to align the stack to 16 bytes before all 'call'
instruction not 8. Otherwise FPU (GP) exception might be raised later on
because of unaligned FPU accesses.

issue #3365
 2019-05-29 10:20:52 +02:00
Emery Hemingway
fc68e97e4a Sculpt: redirect "nic_drv" binary to "ipxe_nic_drv" 
Ref #2190
Fix #3374
 2019-05-29 10:20:52 +02:00
Alexander Boettcher
b3a9ad2215 vbox5: fpu support via new vm_session 
Issue #3111
 2019-05-29 10:20:52 +02:00
Alexander Boettcher
ab6315d6b4 vm_session: add fpu state for x86 
Issue #3111
 2019-05-29 10:20:52 +02:00
Alexander Boettcher
5673c163fb vbox5: use vm_session interface 
Issue #3111
 2019-05-29 10:20:52 +02:00
Alexander Boettcher
07ee9654e4 vm_session: consume spurious wakeups on nova 
Issue #3111
 2019-05-27 14:53:32 +02:00
Alexander Boettcher
9f70084524 base: add vm session API recipe 
Issue #3111
 2019-05-27 14:53:32 +02:00
Sebastian Sumpf
da17f2cbd3 hw: eager FPU switching for x86_64 
Since gcc 8.3.0 generates SSE instructions into kernel code, the
kernel itself may raise FPU exceptions and/or corrupt user level FPU
contexts thereby. Both things are not feasible, and therefore, lazy FPU
switching becomes a no go for base-hw because we cannot avoid FPU
instructions because of the entanglement of base-hw, base, and the tool
chain (libgcc_eh.a).

issue #3365
 2019-05-27 14:53:32 +02:00
Norman Feske
34a711b439 tool/parse_cxx: const var decl, initializers 
This patch handles a few corner cases that appear in the headers of
Genode 19.05, in particular the new client-side block API in
'block_session/connection.h'.
 2019-05-27 14:53:32 +02:00
Christian Helmuth
b359ec14a8 autopilot: align output for wide board names 2019-05-27 14:53:32 +02:00
Alexander Boettcher
68d5293f01 platform_drv: adjust to g++ 8.3.0 
Issue #3307
 2019-05-27 14:53:31 +02:00
Alexander Boettcher
91146a3a70 acpi_drv: adjust to g++ 8.3.0 
Issue #3307
 2019-05-27 14:53:31 +02:00
Norman Feske
405a9d2144 Refinements for updated "Genode Foundations" book 
- Improve API descriptions
- Remove obsolete Xml_node::value method (fixes #3323)
- Follow coding style 'const char' -> 'char const'
- Avoid '>>' when nesting templates (limitation of parse_cxx)
 2019-05-27 14:52:53 +02:00
Christian Helmuth
6c42bd4dd3 libc: use non-anonymous 'operator new' and destroy() 
This removes implementations of and also references to anonymous new and
delete operators from the libc implementation. As allocators for
new/delete Libc::Allocator instances are used, which (paradoxically) map
to libc malloc/free.
 2019-05-27 14:52:53 +02:00
Christian Helmuth
9288fe63ad arg_string: return default value on parsing error 2019-05-27 14:52:53 +02:00
Pirmin Duss
bd045c65a2 util/string.h: suppress case fallthrough warnings 2019-05-27 14:52:53 +02:00
Johannes Kliemann
35b1440c97 ada-runtime: update to GCC 8.3.0 
ref #3362
fixes #3371
 2019-05-27 14:52:53 +02:00
Martin Stein
ba9ffc6243 nic_dump: do not depend on libports repo 
Fixes #3372
 2019-05-27 14:52:52 +02:00
Martin Stein
95ece89cf8 tool/depot: improve handling of missing ports 
* The extract tool determines and reports all missing ports at once.
* The extract tool automatically prepares all missing ports if PREPARE_PORTS=1.
* The missing_ports tool prints a list of missing ports for given archives.

Fixes #3353
 2019-05-27 14:52:52 +02:00
Stefan Kalkowski
6d8d6b5552 hw: disable alignment checking at earliest 
Fix #3370
 2019-05-27 14:52:52 +02:00
Sebastian Sumpf
f18285205c hw: enable FPU during CPU startup on x86 
Also disable TS (task switch) flag in cr0 during kernel initialization,
so FPU faults are not raised. This became necessary since GCC lately
aggressively generates FPU instructions at arbitrary places and also at
early kernel-bootstrapping stages.

fixes #3365
 2019-05-27 14:52:52 +02:00
Christian Helmuth
e60b597af5 linux: 32-bit compatibility with GCC versions before 7 
GCC version 7 and above generate calls to __divmoddi for 64bit integer
division on 32-bit. Unfortunately, libgcc liberaries of older compilers
lack this symbol and are still in use by Debian/Ubuntu LTS at least.
 2019-05-27 14:52:52 +02:00
Christian Helmuth
96b29c6f8a Update lwip_lx test (vfs plugin, ld="no") 2019-05-27 14:52:52 +02:00
Christian Prochaska
a5d6cbf44d tool_chain: strip gdb 
Issue #3307
 2019-05-27 14:52:52 +02:00
Martin Stein
79d1c4f83a nic_router_flood: adapt test timeout 
Fixes #3368
 2019-05-27 14:52:52 +02:00
Christian Prochaska
4dd9d5eb6c dde_zircon: fix compile error with GCC 8.3.0 
Fixes #3366
 2019-05-27 14:52:52 +02:00
Alexander Boettcher
753e78d122 seoul: adjust to g++ 8.3.0 
Issue #3307
 2019-05-27 14:52:52 +02:00
Alexander Boettcher
b50e54b0a8 nova: adjust to g++ 8.3.0 
Issue #3307
 2019-05-27 14:52:52 +02:00
Christian Prochaska
7fadfbbd9f Treat 'implicit fallthrough' errors as warnings 
After fixing the warnings, the compiler option should be removed.

Issue #3307
 2019-05-27 14:52:52 +02:00
Christian Prochaska
b4649d84ee ada-runtime: adjustments for Genode tool chain 19.05 
Issue #3307
 2019-05-27 14:52:52 +02:00
Christian Prochaska
eb4d431e76 stdcxx: update to version 8.3.0 
Issue #3307
 2019-05-27 14:52:52 +02:00
Christian Prochaska
8e2e4374f5 sanitizer: update to version 8.3.0 
Issue #3307
 2019-05-27 14:52:52 +02:00
Christian Prochaska
720919bc14 gcov: update to version 8.3.0 
Issue #3307
 2019-05-27 14:52:52 +02:00
Christian Prochaska
61140380ee base: update 'ld' symbols for tool chain 19.05 
Issue #3307
 2019-05-27 14:52:52 +02:00
Sebastian Sumpf
a8d856fb65 ldso: dynamic linking support for ARM 64-bit 
* added relocation support
* added assembler invocation path for jump slot relocations

fixes issue #3260
 2019-05-27 14:52:52 +02:00
Sebastian Sumpf
7dc875e8c7 base: dynamic linking support of crt0.s on ARM 64-bit 
* added global offset table relative loading for global symbols
* removed 'initial_sp' and 'initial_x0' because they are currently not
  used on this platform. If required they are easy to resurrect.

issue #3260
 2019-05-27 14:52:51 +02:00
Stefan Kalkowski
f0d28eeca7 foc: add support for Raspberry Pi 3 
Ref #3260
 2019-05-27 14:52:51 +02:00
Stefan Kalkowski
71a48c0a26 base: add initial support for ARM 64-bit 
Ref #3260
 2019-05-27 14:52:51 +02:00
Stefan Kalkowski
c98597a2c0 base: increase initial stack 
Ref #3260
 2019-05-27 14:52:51 +02:00
Christian Prochaska
6af3899bcb Enable C++17 by default 
Issue #3307
 2019-05-27 14:52:51 +02:00
Christian Prochaska
74260c96bf tool_chain: integrate 'ali2dep' tool 
Fixes #3361
 2019-05-27 14:52:51 +02:00
Stefan Kalkowski
fc922d263c tool_chain: add aarch64 compiler target 
Ref #3260
 2019-05-27 14:52:51 +02:00
Sebastian Sumpf
36f90e57f3 riscv: tool chain 19.05 
issue #3273
 2019-05-27 14:52:51 +02:00
Christian Prochaska
f369da741d tool_chain: update GDB to version 8.2.1 
Issue #3307
 2019-05-27 14:52:51 +02:00
Christian Prochaska
1720d2d86d tool_chain: update GCC to version 8.3.0 
Issue #3307
 2019-05-27 14:46:54 +02:00
Christian Prochaska
96ed3c8db0 tool_chain: update binutils to version 2.32 
Issue #3307
 2019-05-27 14:46:54 +02:00
Christian Prochaska
fe878e65de base: add copy constructor to 'Genode::Session_label' 
Fixes #3333
 2019-05-27 14:46:54 +02:00
Josef Söntgen
d223539165 wifi_drv: ignore socket flags on socket creation 
The libc features support for SOCK_CLOEXEC now which gets set via
the type argument in 'libnl'. Since we are only interested in the
actual type, i.e. if it is SOCK_RAW, just ignore the flags.

Issue #3289.
 2019-05-27 14:46:54 +02:00
Sebastian Sumpf
a71253fa58 ieee754: adjust to libc/libm update 
* Adjust expected pow(-1/1, inf/-inf/nan) output to current libm
  behavior (as in FreeBSD, glibc, and OpenLibm) and return 1 on
  x86_64/x86_32/arm

* Add 'double float' outputs for ARM which are defined to float

issue #3289
 2019-05-27 14:46:54 +02:00
Christian Helmuth
6db43d2c19 libc_noux target depends on POSIX runtime 
This prevents

  error copying "bin/libm.lib.so": no such file or directory
  error copying "bin/posix.lib.so": no such file or directory

when running "make run/noux" from an empty build directory.
 2019-05-27 14:46:54 +02:00
Christian Prochaska
4fc3eca4aa base-hw: fix compile errors with GCC 8.3.0 
Fixes #3326
 2019-05-27 14:46:54 +02:00
Stefan Kalkowski
5c77ebb1fb hw: factor out x86 specific bootinfo 
Ref #3326
 2019-05-27 14:46:54 +02:00
Stefan Kalkowski
054df95ea4 hw: unify board definitions of bootstrap/core 
Ref #3326
 2019-05-27 14:46:54 +02:00
Martin Stein
8eecb39792 test/timeout: configurable fast-polling buffers 
On some platforms (foc+pbxa9, hw+imx53_qsb_tz, hw+rpi) the default buffer size
is to much for the RAM available on the board. Thus, decrease the buffer size
and therefore the number of polls for these platforms only.

Fixes #3354
 2019-05-27 14:46:54 +02:00
Christian Prochaska
8af81668ea rump_fs: fix run time errors when built with GCC 8.3.0 
Fixes #3360
 2019-05-27 14:46:54 +02:00
Christian Prochaska
c9b5dcafbd vbox: save FPU state in 'EPT violation' exit handler 
Fixes #3359
 2019-05-27 14:46:54 +02:00
Christian Prochaska
f596595c98 vbox: use 'Abi::stack_align()' for exit handler stack 
This ensures proper stack alignment for FPU instructions on x86_64.

Fixes #3358
 2019-05-27 14:46:53 +02:00
Christian Prochaska
3fa994a7a4 dde_bsd: update 'platform_execute()' functions based on libc 
This ensures proper stack alignment for FPU instructions on x86_64.

Fixes #3357
 2019-05-27 14:46:53 +02:00
Christian Prochaska
c7d9df6350 lx_kit: update 'arch_execute()' functions based on libc 
This ensures proper stack alignment for FPU instructions on x86_64.

Fixes #3356
 2019-05-27 14:46:53 +02:00
Christian Prochaska
c2c33d6808 base: save FPU registers in '_jmp_slot' function (x86_64) 
Issue #3355
 2019-05-27 14:46:53 +02:00
Christian Prochaska
9923a1bf50 ports: fix 'seoul' compile errors with GCC 8.3.0 
Fixes #3352
 2019-05-27 14:46:53 +02:00
Emery Hemingway
863654d188 Libc: update port to Freebsd 12 
Fix #3289
 2019-05-27 14:46:53 +02:00
Emery Hemingway
777d92f6de Port of OpenLibm 
Replace the FreeBSD libm with OpenLibm, which is easier to port.
OpenLibm is used by Mirage's freestanding Ocaml runtime (sin POSIX).

https://openlibm.org/

Ref #3289
 2019-05-27 14:46:53 +02:00
Christian Prochaska
382371d9e4 nova: remove '-Wabi' compiler option 
Fixes #3351
 2019-05-27 14:46:53 +02:00
Christian Prochaska
3556a40f81 seoul: adjustments to compile with GCC 8.3.0 
Issue #3352
 2019-05-27 14:46:53 +02:00
Emery Hemingway
4caffd79db Ethernet multicast support at nic_bridge 
Fix #3282
 2019-05-27 14:46:53 +02:00
Stefan Kalkowski
3460444d84 os: name tz_vmm unambigously (ref #2190) 
Moreover, express current requirement to hw API, which was missing until now.

Originally, in the board-specific build directory of imx53_qsb the
"KERNEL" variable was preset with "hw". Therefore, it was not perceived
that this dependency is required.

Ref #3316
 2019-05-27 14:46:53 +02:00
Alexander Boettcher
faee97dd1e sel4: let seoul-kernelbuild.run succeed 
Issue #3111
 2019-05-27 14:46:53 +02:00
Alexander Boettcher
c6ec2c1dd7 libc_terminal plugin: increase stack 
Issue #3111
 2019-05-27 14:46:53 +02:00
Christian Prochaska
92c314d3c8 intel_gpu_drv: fix compile error with GCC 8.3.0 
Fixes #3332
 2019-05-27 14:46:53 +02:00
Christian Prochaska
934ada72fa nova: remove deprecated 'register' keyword 
Fixes #3350
 2019-05-27 14:46:53 +02:00
Christian Prochaska
05a382b1a8 ports: fix 'noux' compile error with GCC 8.3.0 
Fixes #3348
 2019-05-27 14:46:53 +02:00
Christian Helmuth
0777d16e78 Revert "ieee754: disable test for muen (fix #3305)" 
This reverts commit 918b9a9fa4.

The Muen debug console buffer was increased by the recent update, which
alleviates the issue with many log messages as in the ieee754 test.
 2019-05-27 14:46:53 +02:00
Adrian-Ken Rueegsegger
d131e537e9 Update Muen port 
- Drop unnecessary patch
- Improved build speed/parallelization
- Increased log channel size
- Fix path in base-hw/Muen documentation
 2019-05-27 14:46:53 +02:00
Christian Prochaska
37ff9e1e23 gems: fix 'sculpt_manager' compile errors with GCC 8.3.0 
Fixes #3347
 2019-05-27 14:46:52 +02:00
Josef Söntgen
c8c354d1e6 fetchurl: add progress timeout handling 
The component will now abort an ongoing download attempt if it stalls
for given amount of time, the default is 10 seconds.

Fixes #3346.
 2019-05-27 14:46:52 +02:00
Christian Prochaska
4e6216bced libports: fix 'qt5_webcore' compile error with GCC 8.3.0 
Fixes #3345
 2019-05-27 14:46:52 +02:00
Christian Prochaska
5b232df503 libports: fix 'solo5' compile error with GCC 8.3.0 
Fixes #3344
 2019-05-27 14:46:52 +02:00
Stefan Kalkowski
a1e70b9ba4 kernel: differentiate board-specific components 
Components like kernel, core, and bootstrap that are built for a
specific board need to reside inside the same architectural dependent
build directory. For instance there are sel4, foc, and hw kernel builds
for imx6q_sabrelite and imx7d_sabre, which have to reside inside the same
arm_v7 build directory.
This commit names those components explicitely, and adapts the run-tool to it.

Fix #3316
 2019-05-27 14:46:52 +02:00
Stefan Kalkowski
c43267dbaa tool: deprecate board-specific build directories 
* Introduces BOARD variable to determine actual board
* Removes formerly deprecated kernel-specific build directories

The following boards are available:

arm_v6: rpi
arm_v7a: arndale, imx53_qsb, imx53_qsb_tz, imx6q_sabrelite, imx7d_sabre,
         nit6_solox, odroid_x2, odroid_xu, panda, pbxa9, usb_armory,
         wand_quad, zynq_qemu
x86_64: pc, linux, muen
x86_32: pc, linux
riscv:  spike

Ref #3316
 2019-05-27 14:46:29 +02:00
Christian Prochaska
02afb04b7d base: fix 'test-sanitizer' compile error with GCC 8.3.0 
Fixes #3343
 2019-05-16 13:11:03 +02:00
Christian Prochaska
e9fcbace61 libports: fix 'extract' compile errors with GCC 8.3.0 
Fixes #3342
 2019-05-16 13:11:03 +02:00
Christian Prochaska
2b21f41495 demo: fix 'mini_c' compile errors with GCC 8.3.0 
Fixes #3341
 2019-05-16 13:11:03 +02:00
Josef Söntgen
dd6bd0f880 sequence: add keep-going feature 
When enabling the 'keep_going' config attribute, the component will
carry on in case a child exited with an error. In addition, if the
'restart' attribute is set it will start executing the children from
the beginning.
 2019-05-16 13:11:03 +02:00
Martin Stein
1583782446 Revert "nic_router_flood: reworked to stress/analyze more" 
This reverts commit ae55954919.
 2019-05-16 13:11:03 +02:00
Christian Helmuth
22af4436f7 nova: log sp/bp on unresolvable exception 2019-05-16 13:11:03 +02:00
Martin Stein
8fb0d668e0 heap: fix exception handling in _allocate_dataspace 
Previously, only Invalid_dataspace, Region_conflict, and Out_of_ram were
handled for both allocate and attach with the same handlers. However,
both operations can also throw Out_of_caps and for all exceptions during
attach, the dataspace must be freed again whereas this is not the case
when the exception occured during allocate.

Issue #2953
 2019-05-16 13:11:02 +02:00
Sebastian Sumpf
467b96abf4 ldso: lazy binding support for RISC-V 
* added assembler invocation path for jump slot relocations
* fix GOT initialization (jmp_slot pointer goes to GOT[0] not GOT[2] on
  RISC-V)

Fixes #3339
 2019-05-16 13:11:02 +02:00
Christian Helmuth
4f0b17a4dc dde_ipxe: update Intel NIC driver to latest upstream 
This patch is motivated by sporadic hangs during link down/up on i219
NICs handling and the fix implemented upstream in

  https://git.ipxe.org/ipxe.git/commit/546dd51de8459d4d09958891f426fa2c73ff090d

Issue #1220
 2019-05-16 13:11:02 +02:00
Christian Helmuth
a633b5e36e drivers_interactive-pc: quotas (ps2_drv, input_filter) 2019-05-16 13:11:02 +02:00
Christian Helmuth
ea954e7e15 demo: support more platforms in test / drivers RAM 2019-05-16 13:11:02 +02:00
Christian Helmuth
3033496fa4 run: honor DEPOT_DIR for --depot-auto-update 
Issue #3270
 2019-05-16 13:11:02 +02:00
Stefan Kalkowski
13fb51eecf hw: remove implicit SoC-specific include path 
Fix #3336
 2019-05-16 13:11:02 +02:00
Christian Prochaska
9c17c83bf1 os: fix 'test-cpu' compile error with GCC 8.3.0 
Fixes #3335
 2019-05-16 13:11:02 +02:00
Christian Prochaska
2191ff656e os: fix 'test-block_request_stream' compile error with GCC 8.3.0 
Fixes #3334
 2019-05-16 13:11:02 +02:00
Christian Prochaska
2aeb1a70ea os: remove 'register' keyword from 'pixel_rgba.h' 
Fixes #3331
 2019-05-16 13:11:02 +02:00
Christian Prochaska
aa63628536 pistachio: fix compile error with GCC 8.3.0 
Fixes #3330
 2019-05-16 13:11:02 +02:00
Christian Prochaska
5a2e7a8d66 base-sel4: fix compile errors with GCC 8.3.0 
Fixes #3328
 2019-05-16 13:11:02 +02:00
Christian Prochaska
a41d46e193 base-pistachio: fix compile error with GCC 8.3.0 
Fixes #3327
 2019-05-16 13:11:02 +02:00
Christian Prochaska
940ba9ba95 base-nova: fix compile errors with GCC 8.3.0 
Fixes #3325
 2019-05-16 13:11:02 +02:00
Christian Prochaska
96627df4d4 base: fix cxx library compile errors with GCC 8.3.0 
Fixes #3322
 2019-05-16 13:11:02 +02:00
Christian Prochaska
47a2ad604c base: fix xml_node test compile error with GCC 8.3.0 
Fixes #3324
 2019-05-16 13:11:02 +02:00
Alexander Boettcher
70e0514a02 seoul: handle late timeouts 
either
- due to poor signal performance of base platform
- due to being to less prioritized
- due to schedule overload
- due to using time sources of different physical CPUs

Issue #3111
 2019-05-16 13:11:02 +02:00
Stefan Kalkowski
9135be8d5f fb_bench: disable automated test for imx7d_sabre 
This platform hasn't a fb_drv yet.
 2019-05-16 13:11:01 +02:00
Stefan Kalkowski
c5fee20286 sel4: build elfloader on demand (fix #3329) 2019-05-16 13:11:01 +02:00
Stefan Kalkowski
817eb4f23c okl4: create elfweaver tools on demand (ref #3329) 2019-05-16 13:11:01 +02:00
Stefan Kalkowski
4fa34190de platform_drv: check acpi ability by platform_info 
Instead of retieving the information about the underlying platform from
the configuration, check the running kernel from the platform_info. This
commit removes the undocumented "acpi" config attribute.
 2019-05-16 13:11:01 +02:00
Stefan Kalkowski
be053ed257 depot: add recipe for drivers_interactive-muen 2019-05-16 13:11:01 +02:00
Stefan Kalkowski
ae3a6fe270 depot: add recipe for drivers_nic-muen pkg 2019-05-16 13:11:01 +02:00
Stefan Kalkowski
da498af74e imx7d_sabre: disable automated network test 
Until there is no network driver present for this board it is needless to
let the tests fail each night.
 2019-05-16 13:11:01 +02:00
Stefan Kalkowski
1e986fade8 os: name pbxa9 ps2_drv explicitely (ref #2190) 2019-05-16 13:11:01 +02:00
Stefan Kalkowski
d9143f805e os: name rtc_drv unambigously (ref #2190) 2019-05-16 13:11:01 +02:00
Stefan Kalkowski
2e68fae2ec os: name ahci_drv unambigously (ref #2190) 2019-05-16 13:11:01 +02:00
Stefan Kalkowski
d18d2d0d9b os: name sd_card_drv unambigously (ref #2190) 2019-05-16 13:11:01 +02:00
Stefan Kalkowski
4e6b571a36 os: rename uart_drv unambigously (ref #2190) 2019-05-16 13:11:01 +02:00
Stefan Kalkowski
e6f83d4df2 depot: add recipe for drivers_interactive-rpi 
Ref #2190
 2019-05-16 13:11:01 +02:00
Stefan Kalkowski
a623a66019 depot: add recipe for drivers_interactive-imx53_qsb 
Ref #2190
 2019-05-16 13:11:01 +02:00
Stefan Kalkowski
35e73b1a2d os: name imx53 input_drv unambigously (ref #2190) 2019-05-16 13:11:01 +02:00
Stefan Kalkowski
8d1cfce15e os: name fb_drv unambigously (ref #2190) 2019-05-16 13:11:01 +02:00
Stefan Kalkowski
4550056de7 run: use drivers_interactive-* pkg where possible 
Ref #2190
 2019-05-16 13:11:01 +02:00
Johannes Schlatow
0451d3bbed Simple cache performance test 
ref #3321
 2019-05-16 13:11:00 +02:00
Johannes Schlatow
9097c80269 zynq: improve nic driver error handling 
also be more verbose about detected errors

fixes #3320
 2019-05-16 13:11:00 +02:00
Christian Prochaska
739317a83f noux: support non-blocking pipes 
Needed for 'noux_gdb.run' with newer gdb versions.

Fixes #3319
 2019-05-16 13:11:00 +02:00
Alexander Boettcher
92510af9d4 foc: working VM session support for AMD 
- kernel: propagate cr0 to VMM on exit in nested paging case
- kernel: disable forceful VM exit on task switch
- vm_session: adjust to kernel changes

Issue #3111
 2019-05-16 13:11:00 +02:00
Alexander Boettcher
b3f288c035 seoul: avoid corrupted cr0 and GP on debug wrmsr 
- fix bug in instruction emulator on clts - mark cr0 as changed
- don't cause GP on debug control wrmsr

Issue #3111
 2019-05-16 13:11:00 +02:00
Emery Hemingway
761e312219 Vbox: compile VMM/VMMR3/VMReq permissive 
Ref #3289
 2019-05-16 13:11:00 +02:00
Christian Prochaska
c0f03a28e9 gdb_monitor: use terminal VFS plugin 
... instead of the deprecated libc terminal plugin.

Fixes #3318
 2019-05-16 13:11:00 +02:00
Emery Hemingway
d2ab699cd5 Remove Post_signal_hook from Entrypoint 
The Post_signal_hook mechanism has been completely replaced by
Io_progress_handler and can be removed.

Ref #3132
Fix #3302
 2019-05-16 13:11:00 +02:00
Stefan Kalkowski
5d4064fed4 depot: add recipe for drivers_nic-pbxa9 
Ref #3180
 2019-05-16 13:11:00 +02:00
Stefan Kalkowski
5182224c2b depot: add recipe for drivers_nic-zynq 
Ref #3180
 2019-05-16 13:11:00 +02:00
Stefan Kalkowski
8dd618d67f depot: add recipe for drivers_nic-imx6q_sabrelite 
Ref #3180
 2019-05-16 13:11:00 +02:00
Stefan Kalkowski
82693ba5b5 depot: add recipe for drivers_nic-imx53_qsb 
Ref #3180
 2019-05-16 13:11:00 +02:00
Stefan Kalkowski
ff031d792c depot: add recipe for driver_nic-rpi 
Ref #3180
 2019-05-16 13:11:00 +02:00
Stefan Kalkowski
5b5e8a9bd4 depot: add recipe for drivers_nic-linux 
Ref #3180
 2019-05-16 13:11:00 +02:00
Stefan Kalkowski
4fdbb1b1ad depot: add recipe for linux_nic_drv 
Ref #3180
 2019-05-16 13:11:00 +02:00
Stefan Kalkowski
bf5a631a14 dde_linux: name usb_drv unambigously 
* Make package buildable for ARM too
* Move usb library to src targets for explicitly named targets
* adapt remaining run-scripts to use the correctly named usb drivers

Ref #2190
 2019-05-16 13:11:00 +02:00
Stefan Kalkowski
d5104aca05 dde_linux: make rpi usb drivers kernel independent 
Ref #3180
 2019-05-16 13:11:00 +02:00
Stefan Kalkowski
d00bcac941 dde_linux: exclude non-base API from fec_nic_drv 
Ref #3180
 2019-05-16 13:11:00 +02:00
Stefan Kalkowski
5bc498c812 dde_linux: name FEC nic_drv unambigously 
Ref #2190
 2019-05-16 13:11:00 +02:00
Stefan Kalkowski
312499a1ef os: name lan9118 nic driver explicitly 
* it is not dependent on pbxa9 anymore, but configureable
* rename it to lan9118_nic_drv

Ref #2190
 2019-05-16 13:11:00 +02:00
Stefan Kalkowski
dcc28b65cb run: use driver_nic-* pkg where possible (fix #3180) 2019-05-16 13:10:06 +02:00
Stefan Kalkowski
b6d14d9960 dde_ipxe: name ipxe nic_drv unambigously 
Ref #2190
 2019-05-16 12:52:59 +02:00
Stefan Kalkowski
7c1e3c84ba os: name gpio_drv unambigously (ref #2190) 2019-05-16 12:52:59 +02:00
Stefan Kalkowski
911f4ada0a os: make platform_drv package ready for ARM 
* Make target binaries independent of board SPECS
* Name binaries of one architecture unambigously
* Extend include path to match board specifics
* Adapt run-scripts to use the right binary

Ref #2190
Ref #3180
 2019-05-16 12:52:59 +02:00
Stefan Kalkowski
c55663b923 os: extend regulator session api package 
Ref #2190
Ref #3180
 2019-05-16 12:52:59 +02:00
Stefan Kalkowski
cac0f44194 os: extend platform_session api package 
Ref #2190
Ref #3180
 2019-05-16 12:52:59 +02:00
Christian Prochaska
5bc9082fb7 noux_terminal_fs.run: increase noux capability quota 
Fixes #3309
 2019-05-16 12:52:59 +02:00
Emery Hemingway
62a4d1de0e Remove Nim tests 
Nim is now built independently of the Genode repository using an SDK and
Nimble.

Fix #3300
 2019-05-16 12:52:59 +02:00
Ben Larson
5f0aa16184 platform_drv/x86: fix report for many PCI devices 
When there are too many PCI devices, the Expanding_reporter regenerates
the report. However, this doesn't reset the BDF counter used to iterate
over the devices. This results in starting the new report after the PCI
device that triggered the report buffer overflow. This commit fixes the
issue by putting the BDF counter initialization inside the lambda
function used to generate the report.

Fixes #3317
 2019-05-16 12:52:59 +02:00
Christian Helmuth
ba51800b31 depot: update recipe hashes 2019-05-06 16:15:27 +02:00
Stefan Kalkowski
b495799d31 sel4: change ELF loader link address on imx7d 
Fix #3311
 2019-05-06 16:15:27 +02:00
Alexander Boettcher
c0a00019c0 foc/sel4: place vcpu thread on same cpu as ep 
Issue #3111
 2019-05-06 16:15:27 +02:00
Alexander Boettcher
85d98195d5 top: show priority and quota 
Issue #3192
 2019-05-06 16:15:27 +02:00
Emery Hemingway
3e848dff10 solo5: update bindings 
Update the Solo5 upstream to accomadate changes to the Block session and
stack protection support in the base library.

Ref #3275
Ref #3283
 2019-05-06 16:15:27 +02:00
Stefan Kalkowski
b85071174a base: wait a bit at the end of the smp test 
Fix #3306
 2019-05-06 16:15:27 +02:00
Stefan Kalkowski
918b9a9fa4 ieee754: disable test for muen (fix #3305) 2019-05-06 16:15:26 +02:00
Stefan Kalkowski
089cb0f536 foc: change core link address for imx7d_sabre 
Fix #3298
 2019-05-06 16:15:26 +02:00
Stefan Kalkowski
f42c21f16b platform_info: add kernel information (fix #3295) 2019-05-06 16:15:26 +02:00
Norman Feske
c38c80fd43 menu_view: midway re-targeting of geometry motion 
This patch fixes the corner case where an animated geometry changes its
destination mid-way while an animation is already in progress. The
'_trigger_animated_geometry' method used to back out early in this case,
which was intended as an optimization.

Fixes #3296
 2019-05-06 16:15:26 +02:00
Norman Feske
3f8dfa346c sculpt_manager: remove double name attributes 
The 'name' attribute was generated by both the call of 'gen_named_node'
and 'Hoverable_item::gen_button_attr'. Use only the former.
 2019-05-06 16:15:26 +02:00
Alexander Boettcher
05fa063068 vm_session: support to trace vCPU 
Issue #3111
 2019-05-06 16:15:26 +02:00
Christian Prochaska
dca144f2ee noux: fix noux_gdb run script 
Fixes #3287
 2019-05-06 16:15:26 +02:00
Christian Prochaska
4672fdb9f0 gdb_monitor.run: remove broken stderr redirection 
Fixes #3290
 2019-05-06 16:15:26 +02:00
Stefan Kalkowski
d6332f1767 zynq: make nic driver independent of hw kernel 
Use uncached ram dataspace for RX and TX communication buffers
instead of cache maintainance functions that are not part of the
Genode API.

Fix #3291
 2019-05-06 16:15:26 +02:00
Stefan Kalkowski
c767f6ccf1 os: explicit cache policy in Nic::Session_component 
To enable the use of uncached DMA buffers as RX and TX communication
buffers in between driver (service) and client, introduce a cache
attribute in the constructor of Nic::Session_component

Ref #3291
 2019-05-06 16:15:26 +02:00
Martin Stein
0b9916cae2 Timer::Connection: simplify curr_time 
With the new fact that plain time values are always 64 bit unsigned, the
timestamp type is never bigger than the plain time type. Therefore, a code path
in the curr_time interpolation that treated this condition is not neccessary
anymore.

Ref #3208
 2019-05-06 16:15:26 +02:00
Martin Stein
181c78d482 timeout: use uint64_t for all plain time values 
This enforces the use of unsigned 64-bit values for time in the duration type,
the timeout framework, the timer session, the userland timer-drivers, and the
alarm framework on all platforms. The commit also adapts the code that uses
these tools accross all basic repositories (base, base-*, os. gems, libports,
ports, dde_*) to use unsigned 64-bit values for time as well as far as this
does not imply profound modifications.

Fixes #3208
 2019-05-06 16:15:26 +02:00
Stefan Kalkowski
e072ee480b depot: add recipe for base-hw-imx7d_sabre 
Fix #3293
 2019-05-06 16:15:26 +02:00
Stefan Kalkowski
eb933af40a depot: add recipe for base-foc-imx7d_sabre 
Fix #3293
 2019-05-06 16:15:26 +02:00
Stefan Kalkowski
6d0a271308 depot: add recipe for base-sel4-imx7d_sabre 
Fix #3293
 2019-05-06 16:15:26 +02:00
Stefan Kalkowski
4bcd9169c0 sel4: add timer for imx7d_sabre 
Fix #3292
 2019-05-06 16:15:26 +02:00
Stefan Kalkowski
95b9e0a0e2 sel4: fix configuration for imx7d sabre 
Ref #3292
 2019-05-06 16:15:26 +02:00
Alexander Boettcher
cc64c43758 vm_session: adjustments to work with seoul vmm 
for foc, nova, sel4

Issue #3111
 2019-05-06 16:15:26 +02:00
Alexander Boettcher
8950de5a89 base: support whole addressable range in allocator 
Issue #3111
 2019-05-06 16:15:26 +02:00
Alexander Boettcher
8840ca96a9 seoul: use vm_session interface 
and remove any dependency on the NOVA kernel interface

Issue #3111
 2019-05-06 16:15:26 +02:00
Alexander Boettcher
35d46dca72 lib/launchpad: add VM session forwarding support 
Issue #3111
 2019-05-06 16:15:25 +02:00
Alexander Boettcher
393643515c base: extend attach of vm_session 
by offset, size, writeable and executable parameter

Issue #3111
 2019-05-06 16:15:25 +02:00
Norman Feske
17fda73ca1 block_tester: update to new block-client API 
- Added 'io_buffer' attribute, default is 4M
- Added 'batch' attribute, specifying the number of jobs used
  in parallel, default is 1 (sequential)
- Removed 'synchronous' attribute (use batch of 1 instead)
- Added 'copy' attribute (default "yes")
- Print number of signals ("triggered")

Issue #3283
 2019-05-06 16:15:25 +02:00
Norman Feske
784206c85c block_session: new client-side API 
This patch equips the 'Block::Connection' with a framework API for the
implementation of robust block-session clients that perform block I/O in
an asynchronous fashion.

An application-defined 'JOB' type, inherited from 'Connection::Job',
encapsulates the application's context information associated with a
block operation.

The lifecycle of the jobs is implemented by the 'Connection' and driven
by the application's invokation of 'Connection::update_jobs'. The
'update_jobs' mechanism takes three hook functions as arguments, which
implement the applications-defined policy for producing and consuming
data, and for the completion of jobs.

Issue #3283
 2019-05-06 16:15:25 +02:00
Norman Feske
976f67eee6 os: make 'Block::Operation' printable 
Issue #3283
 2019-05-06 16:15:25 +02:00
Norman Feske
e5f27b44b5 block_session: turn 'Connection' into template 
As a preparatory step for introducing the new block-client API, we have
to turn the 'Block::Connection' into a class template. The template
argument will be used to tie an application-defined job type to the
block connection.

Issue #3283
 2019-05-06 16:15:25 +02:00
Norman Feske
6c0dd9fe3b block_session: use types from block/request.h 
Issue #3283
 2019-05-06 16:15:25 +02:00
Stefan Kalkowski
7a45867841 vmm_arm: support for i.MX7 Dual SABRE board 
Fix #3285
 2019-05-06 16:15:25 +02:00
Stefan Kalkowski
0e94d7410d noux_tool_chain_auto: enable imx6 and imx7 boards 
Fix #3288
 2019-05-06 16:15:25 +02:00
Stefan Kalkowski
f074954d3d hw: use eret in hyp mode to switch mode 
Instead of using `cps` instruction, use an exception return
instruction to switch from `hyp` mode to `svc` mode.
Otherwise it causes unpredicted behaviour on ARM.

Fix #3284
 2019-05-06 16:15:25 +02:00
Norman Feske
c433f87000 os: batching of packet-stream source operations 
This patch adds support for manually triggering the wakeup of the packet
sink by the source. This way, a packet source becomes able to marshal
batches of submissions or unmarshal batches of acknowledgements before
yielding the control over to the sink.

Issue #3283
 2019-05-06 16:15:25 +02:00
Norman Feske
ea6b1c255e base: add const version of 'Fifo::Element::object' 2019-05-06 16:15:25 +02:00
Alexander Boettcher
35cf804471 base: free up all blocks on avl destruction 
Issue #3111

remove_range may deny to the job on memory pressure or insane ranges,
which ends up in an endless loop when the Avl allocator is in destruction.

Since the Avl gets destructed, solely the memory free up is of importance,
not the correct range adjustments during remove_range.
 2019-05-06 16:15:25 +02:00
Alexander Boettcher
526554741f vm_session: evaluate priority (nova) 
Issue #3111
 2019-05-06 16:15:25 +02:00
Alexander Boettcher
d2229ab381 vm_session: evaluate priority 
Issue #3111
 2019-05-06 16:15:25 +02:00
Alexander Boettcher
d459432c45 init: scale priority of vm_session 
Issue #3111
 2019-05-06 16:15:25 +02:00
Alexander Boettcher
450c8dc149 vm_session: track dataspaces used by attach 
Track the dataspaces used by attach and add handling of flushing VM space
when dataspace gets destroyed (not triggered via the vm_session interface).

Issue #3111
 2019-05-06 16:15:25 +02:00
Alexander Boettcher
169c51d50d base: remove error message in slab block 
Issue #3111
 2019-05-06 16:15:25 +02:00
Alexander Boettcher
d1f37e66eb base: fix _sum_in_range in avl structure 
Handles corner case when addr + size becomes exactly 0. Before the commit
the function returned that sum is not part of the range, which is wrong.

Issue #3111
 2019-05-06 16:15:24 +02:00
Alexander Boettcher
812149ed29 core: add Region_map_detach interface 
Issue #3111
 2019-05-06 16:15:24 +02:00
Norman Feske
593e971121 block_session: SYNC and TRIM as async operations 
This patch removes the blocking Block::Session::sync RPC function and
adds the asynchronous operations SYNC and TRIM to the block session's
packet-stream interface.

Even though the patch adjusts all block components to the interface
change, the components keep the former blocking handling of sync
internally for now because of the design of the 'Block::Driver'
interface. This old interface is not worth changing. We should instead
migrate the block servers step by step to the new
'Block::Request_stream' API.

Fixes #3274
 2019-05-06 16:15:24 +02:00
Norman Feske
e058adc4fe block_session: add client-defined request tag 
The new request tag allows a block-session client to uniquely correlate
acknowledgements with outstanding requests. Until now, this was possible
for read and write operations by taking the value of the request's
packet-stream offset. However, SYNC and TRIM requests do not carry any
packet-stream payload and thereby lack meaningful offset values. By
introducing the notion of a 'tag', we can support multiple outstanding
requests of any type and don't need to overload the meaning of the
'offset' value.

Issue #3274
 2019-05-06 16:15:24 +02:00
Norman Feske
a39474a245 block-request stream: split Operation from Request 
This patch splits the 'Request' definition into smaller types that are
suitable for the client-side API too.

The new 'Operation' type comprises the block operation's type (opcode)
and the operation's arguments (block number, block count).
The former 'Request::operation_defined' is now 'Operation::valid'.
The 'Request' aggregates an 'Operation', which changes its object
layout.

Note that this commit relaxes the bit-precise definition of 'Request' to
facilitate the use of 'unsigned long' where appropriate, in particular
for the request tag (which should correspond to an 'Id_space::Id'). The
originally bit-precise definition was pursued to allow the sharing of
the 'Request' type between SPARK and C++ code. However, it turns out
that defining a native type in each language and a (set of) converting
constructors is a more natural approach.

Issue #3283
 2019-05-06 16:15:24 +02:00
Christian Prochaska
ed246d152b tool_chain: build in <genode-dir>/build/tool_chain-<version> 
Fixes #2608
 2019-05-06 16:15:24 +02:00
Stefan Kalkowski
7f965dad29 vmm_arm: do not use uart driver in automated test 
Fix #3278
 2019-05-06 16:15:24 +02:00
Stefan Kalkowski
0a470e66f2 vmm_arm: uart device model needs to take all chars 
When receiving a terminal signal, the uart device model has to
take all characters out of the stream. Otherwise, characters
might never arrive at the VM. This was not recognized before,
because it was used with a quite slow UART only, which obviously
never achieved to send more characters at once.

Ref #3278
 2019-05-06 16:15:24 +02:00
Stefan Kalkowski
94881e757a test-terminal_expect_send: expect-like helper tool 
A small terminal-client tool, which expects a specific line(-start),
and then sends a specified line to the other side. Optionally,
it prints all received lines to its LOG service.

Ref #3278
 2019-05-06 16:15:24 +02:00
Alexander Boettcher
26796c8f7a nova: update kernel branch 
- sorting of initializer by Christian Prochaska - issue #3253
- leak of FPU register state reported by Julian Stecklina - thanks !
- fix page table synchronization bug - by Julian Stecklina
- add a second dynamic buddy if the memory requirements could not be fulfilled
  by the first dynamic buddy
- disable vtlb peek if EPT/NPT is enabled
 2019-05-06 16:15:16 +02:00
Norman Feske
51f51c18af block/request_stream: rename wakeup_client 
This patch renames 'wakeup_client' to 'wakeup_client_if_needed' to
clarify that the method triggers signals only when needed, not on every
call.

The name 'wakeup_client' is prone to misguide users to call the function
conditionally as an optimization, thereby complicating the code, but to
no effect.

Fixes #3279
 2019-05-03 13:53:12 +02:00
Norman Feske
34fd5f626a test/block_request_stream: private Request_stream 
Avoid public inheritance to make the code simpler to understand.

Issue #3279
 2019-05-03 13:53:12 +02:00
Norman Feske
bbe3ee8dc5 block_session: server-defined payload alignment 
This patch replaces the formerly fixed 2 KiB data alignment within the
packet-stream buffer by a server-defined alignment. This has two
benefits.

First, when using block servers that provide small block sizes like 512
bytes, we avoid fragmenting the packet-stream buffer, which occurs when
aligning 512-byte requests at 2 KiB boundaries. This reduces meta data
costs for the packet-stream allocator and also allows fitting more
requests into the buffer.

Second, block drivers with alignment constraints dictated by the
hardware can now pass those constraints to the client, thereby easing
the use of zero-copy DMA directly into the packet stream.

The alignment is determined by the Block::Session_client at construction
time and applied by the Block::Session_client::alloc_packet method.
Block-session clients should always use this method, not the 'alloc_packet'
method of the packet stream (tx source) directly. The latter merely
applies a default alignment of 2 KiB.

At the server side, the alignment is automatically checked by
block/component.h (old API) and block/request_stream.h (new API).

Issue #3274
 2019-05-03 13:53:12 +02:00
Norman Feske
2208220c12 block session: remove Block::Session::Operations 
This patch modernizes the 'Block::Session::info' interface. Instead of
using out parameters, the 'init' RPC function returns a compound 'Info'
object now. The rather complicated 'Operations' struct is replaced by
a 'writeable' attribute in the 'Info' object.

Fixes #3275
 2019-05-03 13:53:12 +02:00
Norman Feske
10c567daee base: add 'aligned' function to util/misc_math.h 
This function simplifies the sanity checking of values that are expected
to be aligned, e.g., data offsets within packet streams.
 2019-05-03 13:31:39 +02:00
Christian Helmuth
6ea1179145 depot: update recipe hashes 2019-04-09 12:30:35 +02:00
Christian Helmuth
71a8dcca08 tool/depot: use nullglob on stale archive removal 
nullglob is needed to get an empty shell glob in case no matching binary
archive (path) exists. Otherwise, the original glob string including the
* is returned and used in the for loop.
 2019-04-09 12:30:35 +02:00
Stefan Kalkowski
a147bdf406 hw: silent warning about unknown signal context 
Triggering of an invalidated signal seems to be no real exception,
but something that occurs regularily. Therefore, the kernel warning
is of no use to developers anymore.

Ref #3277
 2019-04-09 12:30:35 +02:00
Stefan Kalkowski
0ca199f89a hw: replace lock-safe log variants in kernel 
As far as possible remove usage of warning/error/log in the kernel,
otherwise the kernel context might try to take a lock hold by a core
thread, which results in a syscall to block.

Fix #3277
 2019-04-09 12:30:35 +02:00
Norman Feske
9e238d624e extract: discharge dependency from timer session 
This patch removes the reliance of the extract tool from the libc's
behavior regarding the access of time and timing.

The extract tool is not expected to need time. However, unfortunately,
libarchive calls the 'time' function unconditionally. By adding a
dummy for 'time', we avoid bothering the libc, which would otherwise
need to obtain a time source.

Issue #3204
 2019-04-09 12:30:35 +02:00
Norman Feske
6d11591d83 test-spark: rename number_of_bits -> object_size 
In the original version, I used 'number_of_bits' because Ada's 'Size
returns the size in bits, not bytes. But the values (for objects) are
always a multiple of 8. On the C++ side, performing size checks at the
granularity of bits is just awkward. The term 'object size' is more
natural.
 2019-04-09 12:30:35 +02:00
Sebastian Sumpf
5937492e93 base-foc: map debug cap through the correct index 
* necessary for enabling the kernel debugging facilities on base-foc
  (outstring, fiasco_tbuf_log_3val and friends)
* disabled by default

related to issue #3260
 2019-04-09 12:30:35 +02:00
Norman Feske
ec70081258 test-ldso: refine log pattern 
By adding a wildcard at the beginning of (expected) error messages, the
test.run tool becomes able to match the lines (ignoring the characters
of the color escape sequence).
 2019-04-09 12:30:35 +02:00
Christian Helmuth
9948a77558 spark: provide rcheck symbols in ABI 2019-04-09 12:30:35 +02:00
Norman Feske
724761565d libsparkcrypto: don't override CUSTOM_ADA_OPT 
CUSTOM_ADA_OPT must not be defined outside <build-dir>/etc/tools.conf
 2019-04-09 12:30:35 +02:00
Martin Stein
9d236e8e03 block/request: fixed enum values 2019-04-09 12:30:35 +02:00
Martin Stein
a9851aca55 block/request: add member 'tag' 
Issue #3274
 2019-04-09 12:30:35 +02:00
Norman Feske
a2743dcaeb Library for the AES-CBC en/decryption of 4K blocks 
The 'aes_cbc_4k' library is simple wrapper around libsparkcrypto to
serve as a backend for storage encryption. It operates on data chunks of
4 KiB and uses AES-CBC while incorporating the block number and the
private key as salt values.
 2019-04-09 12:30:35 +02:00
Pirmin Duss
8b4e2a21e4 e2fsprogs: disarm the mount-point check 
Discovered while creating an ext2 file system on an sd_card (no
partitions, imx6), which failed with an error message that stated the
device is in use. Genode's libc/posix layer has no notion of mount
points.

Fixes #3271
 2019-04-09 12:30:35 +02:00
Pirmin Duss
642c2ab4bc sd_card_drv: increase multiblock transfer timeout 
Disconnecting a client and connecting an other to the sd_card_drv
on imx6 results in a "Completion host signal timed out" error in
the newly connected client.

Fixes #3272
 2019-04-09 12:30:35 +02:00
Alexander Boettcher
93ba870b2d top: show execution times per SC and per EC 
Issue #3192
 2019-04-09 12:30:35 +02:00
Alexander Boettcher
36fac8e22b run/test: support TRACE service 
Issue #3192
 2019-04-09 12:30:35 +02:00
Alexander Boettcher
ae16edf1d6 trace: support more facets of execution time 
- execution time per thread context
- execution time per scheduling context
- quantum and priority

Issue #3192
 2019-04-09 12:30:34 +02:00
Emery Hemingway
66f49e6c42 VFS lwIP: support opening a handle on the root directory 
The "nameserver" file cannot be opened through a VFS File_system client
if the plugin does not support opening the parent directory of
"/nameserver", which would be "/".

Ref #3269
 2019-04-09 12:30:34 +02:00
Emery Hemingway
a635873568 VFS: Replace global response handlers with local handlers 
Replace the I/O response handler that is passed to the VFS at
construction with an object that is dynamically attached to handles.
This object shall also accept read-ready notifications, and plugins are
encouraged to keep handles awaiting ready-ready notifications separate
from handles that await I/O progress.

Replace the use of handle lists in plugins with handle queues, this
makes the code easier to understand and the ordering of notifications to
the application more explicit.

These changes replace the use of the Post_signal_hook from all VFS
plugins, applications must assume that read-ready and I/O notifications
occur during I/O signal dispatch and use an Io_progress_handler at its
entrypoints to defer response until after signal dispatching.

Fix #3257
 2019-04-09 12:30:34 +02:00
Emery Hemingway
e2ff776b35 Print error locations in test-libc_connect 
Ref #3257
 2019-04-09 12:30:34 +02:00
Alexander Boettcher
356526d610 foc: increase USER_BASE_CAP 
due to THREAD_AREA_SLOT increase by

foc/x86: implement vm_session interface

Add additional static assertion check to detect misconfiguration earlier.
 2019-04-09 12:30:34 +02:00
Josef Söntgen
38a10c92d3 pthread: initialize static rwlocks/conds 
Make sure that the rwlock is allocated before a lock operation is
performed. This case occurs if a static rwlock was create by using
PTHREAD_RWLOCK_INITIALIZER. Same goes for PTHREAD_CONDS_INITIALIZER.

Fixes #3262.
 2019-04-09 12:30:34 +02:00
Josef Söntgen
3c8714ed5a dde_bsd: make HDMI/DP HDA check stricter 
Since QEMU might put the audio device at 00:03.0, also check if the
vendor is Intel. Hopefully we do not render HDA on real machines
useless with this changes (so far I have not encountered one).

Fixes #3263.
 2019-04-09 12:30:34 +02:00
Josef Söntgen
3362216b66 os: clear remaining samples in Audio packet 
... instead of potentially producing out-of-bounds write in the source
buffer.

Fixes #3264.
 2019-04-09 12:30:34 +02:00
Josef Söntgen
5ceba11982 os: make Audio_out content filler data arg const 
Issue #3264.
 2019-04-09 12:30:34 +02:00
Josef Söntgen
b9e12d7c23 libc: always set O_RDWR in fcntl on a socket 
For better or worse we have no proper way of handling this right now
but contrib libraries, e.g. glib, use it to determine if they can use
the underlying fd.

Fxies #3265.
 2019-04-09 12:30:34 +02:00
Josef Söntgen
5af0162b3f libc: fix storing TIOCGWINSZ result 
Fixes #3266.
 2019-04-09 12:30:34 +02:00
Norman Feske
a3411c8e96 run: add --depot-auto-update feature 
If enabled, this option triggers the automated management of depot
content according to the needs of a run script.

Fixes #3270
 2019-04-09 12:30:34 +02:00
Norman Feske
d027f12764 depot: update recipe hashes 2019-04-02 09:36:39 +02:00
Norman Feske
4b0b4928f5 tool/depot: rm binary archives if versions collide 
This patch handles version collisions of binary archives. If a binary
archive for an (automatically) updated src-archive version already
exists in the depot, the extract tool removes the binary archive because
its existing content may stem from another src content (a version from a
different topic branch). This ensures that the new version is always
built, not skipped, when 'UPDATE_VERSIONS=1 REBUILD=' is specified.

Fixes #3267
 2019-04-01 19:33:52 +02:00
Alexander Boettcher
ca32c11f4f nitpicker: send mode change on domain update 
Fixes #3259
 2019-04-01 19:33:52 +02:00
Christian Prochaska
38de57be76 tool_chain: make sudo command configurable 
Fixes #3255
 2019-04-01 19:33:52 +02:00
Norman Feske
2b8ea7845f smartcard.run: increase caps for usb_drv 
This is needed for running the test on seL4.
 2019-04-01 19:33:52 +02:00
Alexander Boettcher
483de40c96 base: remove message in sliced_heap.alloc() 
Issue #3111
 2019-04-01 19:33:52 +02:00
Norman Feske
a224d828f6 test.run: integrate content from raw archives 
This patch enables the test.run tool to execute the test-init package.

Issue #3256
 2019-04-01 19:33:52 +02:00
Alexander Boettcher
ef130a3bf9 sel4/x86: implement vm_session interface 
Issue #3111

- enable vt-x in kernel configuration

Kernel patches:

- add unrestricted guest support
- avoid kernel boot failure when vt-x is not available
- avoid nullpointer in kernel when vcpu is not fully setup
- avoid vcpu scheduling bug which causes starvation on same/below prio level
- save efer register correctly from guest
 2019-04-01 19:33:52 +02:00
Alexander Boettcher
b75307b070 foc/x86: implement vm_session interface 
Issue #3111
 2019-04-01 19:33:51 +02:00
Alexander Boettcher
3a1bbfad28 nova: implement vm_session interface 
Issue #3111
 2019-04-01 19:33:51 +02:00
Alexander Boettcher
ae31b761f4 hw: move vmm.run to vmm_arm.run 
Issue #3111
 2019-04-01 19:33:51 +02:00
Alexander Boettcher
f927684eb8 os: add vmm_x86 interface test 
Issue #3111
 2019-04-01 19:33:51 +02:00
Alexander Boettcher
5fa91a1bcc base: add x86 vm_state 
Issue #3111
 2019-04-01 19:33:51 +02:00
Norman Feske
aeb7e7cd7c sculpt manager: prevent double destruction 
The '_currently_constructed' pointer caches the information about which
'Launched_child' is currently configured in the menu. When discarding
the runtime (e.g., when un-using a file system) at this point, this
cached pointer was not invalidated while all 'Launched_child' objects
would be freed (including the currently constructed one). On the next
attempt to construct a new child, the sculpt manager attempted to
destruct the 'Launched_child' referred by the (now outdated)
'_currently_constructed' again.

Fixes #3240
 2019-04-01 19:33:51 +02:00
Stefan Kalkowski
935abb55b7 hw: move src/lib/hw header to src/include/hw 
* Remove bad style of using `src/lib` as include path

Fix #3244
 2019-04-01 19:33:51 +02:00
Alexander Boettcher
00953e39f4 vbox: allocate REM memory dense 
Fixes #2116
 2019-04-01 19:33:50 +02:00
Alexander Boettcher
3b36673e03 seoul: use nitpicker session 
Issue #3196
 2019-04-01 19:33:50 +02:00
Alexander Boettcher
3b36542212 vbox5: use nitpicker session 
Issue #3196
 2019-04-01 19:33:50 +02:00
Josef Söntgen
af97a581e3 bsd_audio_drv: fix runtime for Sculpt CE 
Fixes #3250.
 2019-04-01 19:33:49 +02:00
Stefan Kalkowski
baf815d099 hw: add support for i.MX7 Dual SABRE board 
Fix #3251
 2019-04-01 19:33:49 +02:00
Stefan Kalkowski
8f28f884ee hw: name vm_state header explicitely 
Ref #3251
 2019-04-01 19:33:49 +02:00
Stefan Kalkowski
4c492a3be7 imx7d_sabre: add board for sel4 and foc 
Ref #3251
 2019-04-01 19:33:49 +02:00
Stefan Kalkowski
3d727a403f sel4: update tools to currently used version 9.0.x 
Until now, Genode referenced a fork of the outdated elfloader-tool
to enable bootstrapping of sel4 on ARM platforms. Because the
elfloader is inherently dependent on the used platforms newer
ARM versions supported by the kernel could not be loaded by the
outdated elfloader. This commit uses a fresh fork of the nowadays
used sel4_tools repository.

Ref #3251
 2019-04-01 19:33:49 +02:00
Alexander Boettcher
c19c5ed0a4 wm: fix destroy with invalid handle 
Issue #3232
 2019-04-01 19:33:49 +02:00
Roman Iten
1caa2ff3cb prepare_port: fix race condition in safety rule 
The safe guard Git repository in $(CONTRIB_DIR) may only be initialized
once, regardless of the number of ports to be prepared.

Issue #3235
 2019-04-01 19:33:49 +02:00
Norman Feske
ed33cf08c7 test-fs_tool: make output deterministic 
Issue #3238
 2019-04-01 19:33:48 +02:00
Norman Feske
a67b23eb2d test.run: handle '*' wildcards 2019-04-01 19:33:48 +02:00
Norman Feske
01cff3ce84 sculpt: inform user about incomplete pkg install 
This commit handles the corner case where a package could be installed
successfully but the package's runtime definition is inconsistent with
the content delivered by the package's dependencies, i.e., the <content>
of the runtime file lists ROM modules that do not exist.

With this patch, the '+' menu shows the message "installed but
incomplete" whenever a package is in such a state.

Issue #3241
 2019-04-01 19:33:48 +02:00
Stefan Kalkowski
d6376f8188 rustc: select ARM target correctly (fix #3087) 2019-04-01 19:33:48 +02:00
Christian Helmuth
f0a3377247 Remove obsolete symbols from symbols/ld ABI 2019-04-01 19:33:48 +02:00
Christian Helmuth
9debc0fa4b Mention sel4 kernel as option for imx6q_sabrelite 2019-04-01 19:33:48 +02:00
Christian Helmuth
37a93f53c4 Provide Genode::cache_coherent() consistently 
It got lost on base-foc and also was not put into Genode namespace in
the default implementation.
 2019-04-01 19:33:48 +02:00
Alexander Boettcher
c3fd572a52 init: add report/buffer attribute to xsd 
Issue #3216
 2019-04-01 19:33:48 +02:00
Martin Stein
bcc049ceeb nic_router: manipulate port-forwarding port 
The new configuration attributes <tcp-forward to_port="123" /> and
<udp-forward to_port="123" /> enable manipulating the destination port of
port-forwarded packets.

Fixes #3237
 2019-04-01 19:33:48 +02:00
Martin Stein
3a59f11708 nic_router/nic_bridge: free MAC addresses 
Free a MAC address of a session as soon as the session gets destructed.
Adds also according tests to the autopilot list.

Fixes #2470
 2019-04-01 19:33:48 +02:00
Alexander Boettcher
bafe3c3fc3 vbox5: check memory configuration stricter 
The required memory of the VMM depends on the VM memory configuration (vbox)
and the available screen resolution. Check for insane configurations in the
beginning to avoid late surprises.

Issue #3216
 2019-04-01 19:33:47 +02:00
Alexander Boettcher
ce1f245952 vbox5: free bitmap source during resize 
Fixes #3216
 2019-04-01 19:33:47 +02:00
Emery Hemingway
800b4e44b1 Reduce Vfs::Vfs_handle::Context to empty struct type 
The "Vfs::Vfs_handle" type should not contain any public members that
can be initialized by the VFS internally and by the application, so
remove inheritance from the "Genode::list::Element" class. The VFS
plugins must instead use lists of "Vfs::Vfs_handle" sub-classes, the
lifetime of which are always managed by the plugin.

Ref #3036
 2019-04-01 19:33:47 +02:00
Emery Hemingway
260fc30be3 lxip: consolidate lx timers and timeouts 
Use a single timeout scheduler passed during lxip instantiation for
both timers and 'schedule_timeout' facilites rather than instantiate
two timer sessions and signal handlers. This reduces the library's
capability cost and initialization time.

Fix #2961
 2019-04-01 19:33:47 +02:00
Stefan Kalkowski
3725e91603 hw: implement power-saving kernel lock for ARM smp 
Thanks to former work of Martin Stein this commit finally incorporates a
non-spinning kernel lock on multi-core ARM platforms.

Fix #1313
 2019-04-01 19:33:47 +02:00
Martin Stein
b04a70177b base & arm: simplify and optimize cmpxchg 
The old inline assembly provided two output operands that were afterwards
combined for a return value. However, the second output operand isn't
necessary when using the "Acquiring a Lock" example-code of the ARM manual
"Barrier Litmus Tests and Cookbook". This saves two logical operations
per acquisition try. Additionally better documentation is now provided.

Fixes #1292
 2019-04-01 19:33:47 +02:00
Stefan Kalkowski
e9b3569f44 hw: remove overall cache maintainance from core 
This functionality is only needed in bootstrap now that kernel and
userland share the same address-space.

Fix #2699
 2019-04-01 19:33:46 +02:00
Christian Helmuth
9bbc91bb52 Add missing newline to pipe test 
Original patch by Ben Larson <hungryninja101@gmail.com>.

Fixes #3055
 2019-04-01 19:33:46 +02:00
Emery Hemingway
5efa6d5273 Stack smashing test 
A test to check if -fstack-protector can be enabled and is effective.

Ref #3066
 2019-04-01 19:33:46 +02:00
Emery Hemingway
6819c43a05 Add stack protector storage to base library 
This patch adds the items necessary for building Genode components with
stack protection enabled, but it is not initialized at runtime. They are
provided at the moment as a convenience and do not implement a security
feature.

Fix #3066
 2019-04-01 19:33:46 +02:00
Tomasz Gajewski
35c17ced72 Trivial doc fix for Register_set_plain_access::read 
Fix #3229
 2019-04-01 19:33:46 +02:00
Sebastian Sumpf
1705aab290 seoul: PS2 mouse wheel support 
fixes #3234
 2019-04-01 19:33:46 +02:00
Christian Prochaska
9f50b8897a Update Arora default bookmarks 
Fixes #3198
 2019-04-01 19:33:46 +02:00
Pirmin Duss
1c746df82c fetchurl: update README 
Fix the configuration example in the README.

Fix #3213
 2019-04-01 19:33:46 +02:00
Pirmin Duss
8456ffa231 bomb: prevent Timer-session creation errors 
Fix #3212
 2019-04-01 19:33:46 +02:00
Christian Helmuth
ebe71a12ed depot: update recipe hashes 2019-03-19 11:12:36 +01:00
Norman Feske
0872406e3c News item for Sculpt CE 2019-03-19 11:00:01 +01:00
Norman Feske
cffe847778 Update documentation for Sculpt CE 2019-03-19 11:00:01 +01:00
Norman Feske
4353c9e0f1 os: make Block::Request_stream::Ack noncopyable 
This is a safeguard against accidentally taking the 'Ack' interface as a
value instead of a reference.

The 'Payload' interface should also not be copied (and potentially
stored) because it contains a pointer.
 2019-03-19 11:00:01 +01:00
Norman Feske
6763ac29d9 depot_download_manager: fetchurl watchdog 
If fetchurl the amount of bytes downloaded by fetchurl does not change
for 5 seconds, respawn it.
 2019-03-19 11:00:01 +01:00
Norman Feske
4f99224255 ram_fs: increase max file size on 64 bit to 8 GiB 
Fixes #2315
 2019-03-18 15:57:00 +01:00
Norman Feske
c76f7c0c2a sculpt: allow graph to overlap the log 2019-03-18 15:56:59 +01:00
Norman Feske
806dc5de36 window layouter: improved maximize handling 
It turns out that the commit "window layouter: allow floating apps to
resize" interplays badly with the interactive toggling of the maximize
state of windows. In contrast to the window geometry and stacking, which
is always updated through the rules-feedback mechanism, interactive
changes of the maximize state omitted this loop and instead took a local
shortcut. Because of this shortcut, the maximized geometry eventually
ended up as window size in the window's assign rule. So unmaximizing the
window failed to revert the geometry to its original state.

This patch removes this inconsistency. The maximize state adheres to the
official chain of commands through the rules mechanism now. The state is
now maintained internally without affecting the window's geometry and is
evaluated while generating the window layout only.

As a minor loosely related improvement, this patch prevents the
highlighting of resize handles for non-resizable windows.

Issue #3200
 2019-03-18 15:56:59 +01:00
Alexander Boettcher
104def8e51 acpica: update to sculpt ce 2019-03-18 15:56:59 +01:00
Alexander Boettcher
dc4513f965 sculpt_manager: support system state ROM 2019-03-18 15:56:59 +01:00
Stefan Kalkowski
822a6e7c5f hw_riscv: strictly separate machine and syscall ids 
Fix #3230
 2019-03-18 15:56:59 +01:00
Norman Feske
91197804ac nitpicker: fix destroy with invalid handle 
This patch reworks the 'Session_component::destroy' to cope become
robust against a client-provided invalid view handle. The code did not
consider that 'Handle_registry::has_handle' may throw.

Thanks to Alexander Boettcher for reporting and the initial fix.

Fixes #3232
 2019-03-18 15:56:59 +01:00
Martin Stein
ff2516deb2 hw: fix documentation of Kernel::update_pd 
Add note that the calling thread must not be destroyed while in the syscall.

Fixes #1253
 2019-03-18 15:56:59 +01:00
Martin Stein
a74ae75680 run/nic_router foc x86_64: raise test timeout 
Fixes #2908
 2019-03-18 15:56:59 +01:00
Norman Feske
9438caa6a3 base: never throw in Xml_node::for_each_sub_node 
Fixes #3231
 2019-03-18 15:56:59 +01:00
Norman Feske
d75c5f6722 window layouter: fix glitch after window resize 
This patch improves the transition from an interactive window geometry
change (dragging a window element) to the point where the resulting
new layout rules come into effect. During this short time, no resize
request must be issued because such a resize request would be based on
stale rules.

Fixes #3227
 2019-03-18 15:56:59 +01:00
Norman Feske
baf46db287 Adjust wm -> focus label 
This is a follow-up commit to "Update <provides> info in pkg runtimes",
which adapts the users of the wm pkg to the changed label of the "focus"
nitpicker session.
 2019-03-18 15:56:59 +01:00
Christian Prochaska
67fd77d10a base: catch 'Ipc_error' in 'Expanding_parent_client::exit()' 
If a component is being destroyed just before it calls `exit()` at its
parent, the `exit()` call causes an `Ipc_error` exception, which leads to
an `abort()` loop with repeated error messages, because `abort()` calls
`exit()` too. Catching the exception in `Expanding_parent_client::exit()`
avoids this problem.

Fixes #3228
 2019-03-18 15:56:59 +01:00
Sebastian Sumpf
98e2f91036 qemu-usb: fix multiple device support 
bus or device should be different (not both) when comparing a device to
already present devices. Because of this the second USB device was
marked as existent when the bus matched or the bus did not but the
device number.
 2019-03-18 15:56:59 +01:00
Sebastian Sumpf
326deb14fe sculpt: add global KEY_SCREEN for themed_wm 
fixes switching windows by global key
 2019-03-18 15:56:59 +01:00
Norman Feske
591e9457e6 window layouter: allow floating apps to resize 
This patch gives applications the ability to control the size of their
window whenever the window is floating, not tiled or maximized. See the
comment in the code for the rationale.

Fixes #3200
 2019-03-18 15:56:59 +01:00
Norman Feske
1f3f7282f3 depot_download: limit rate of update-state reports 
The default rate of 100 ms keeps Sculpt too busy because the menu that
displays the percentage values is drawn completely on each update.
Limiting the rate to 1/4 seconds relieves the effect.
 2019-03-18 15:56:59 +01:00
Norman Feske
1ead0ea3a7 sculpt: allow for removal of index files 
By clicking on a yellow checkbox in the depot selection dialog, the
corresponding index files are removed. This way, index files can
be update by removing and downloading them again.

This patch also filters out sculpt-managed components from the graph to
avoid erratic graph-position changes while the '+' menu is open.

Fixes #3193
 2019-03-18 15:56:59 +01:00
Norman Feske
f3d4ee4d4c init: always abandon child on version change 
Without this patch, a version change of an already exited child would not
trigger the restart of the child because the version is evaluated as
late as the child configuration, but only if the child has not exited
yet.

This patch evaluates the version at the earlier stage where the identity
of the child (its name) is checked against the new configuration.

Fixes #3226
 2019-03-18 15:56:58 +01:00
Josef Söntgen
8ccfe4361c part_block: properly indent code 
Issue #3223.
 2019-03-18 15:56:58 +01:00
Josef Söntgen
cb2ebd0bf7 part_block: fix handling of chained EBR's 
Fixes #3223.
 2019-03-18 15:56:58 +01:00
Josef Söntgen
1c2dabc6d7 test-part_block_mbr: update disk image 
The disk image contains an extended partition table now that would
have been parsed incorrectly in the past. While there remove the unused
gpt BLOB (test-part_block_gpt has its own raw archive).

Issue #3223.
 2019-03-18 15:56:58 +01:00
Norman Feske
cac3f3b24e gems: initial version of fs_tool 
The fs_tool component performs file operations according to its
configuration. This initial version implements only the operation
<remove-file> as needed for Sculpt CE.

Issue #3222
Issue #3193
 2019-03-18 15:56:58 +01:00
Norman Feske
73f79ed5b1 gems/vfs.h: add Directory::unlink 
This is needed to accommodate the fs_tool component.

Issue #3222
 2019-03-18 15:56:25 +01:00
Norman Feske
4264de05f9 sculpt: avoid duplicated diagnostic messages 
Related to #3190
 2019-03-18 15:56:25 +01:00
Norman Feske
6528f50bfc sculpt: NIC ready only with IP other than 0.0.0.0 
This patch refines the criterion of when the networking is considered as
ready to use. Until now, any IP reported by the NIC router was taken as
an indicator for connectivity. But as the NIC router reports an IP
0.0.0.0/32 when no network cable is plugged at the uplink, the condition
was too loose.
 2019-03-18 15:56:25 +01:00
Norman Feske
94e2a64f60 Sculpt: defer config dialog until pkg is complete 
Fixes #3189
 2019-03-18 15:56:25 +01:00
Stefan Kalkowski
330692350e hw: introduce non-blocking signal checking 
* Introduces pending_signal syscall to check for new signals for the
  calling thread without blocking
* Implements pending_signal in the base-library specific for hw to use the
  new syscall

Fix #3217
 2019-03-18 15:56:24 +01:00
Stefan Kalkowski
3e70b53165 vfs_lxip: fix return code in blocking accept case 
Ref #3217
 2019-03-18 15:56:24 +01:00
Norman Feske
00eb100114 sculpt: gracefully handle missing depot-user info 
This patch improves the error handling of depot-download manager for the
case where a download is requested but the corresponding software
provider information is absent from the depot. Without this patch, the
update mechanism would get stuck in the failed depot-query step and
won't attempt to perform subsequent download jobs.

Fixes #3224
 2019-03-18 15:56:24 +01:00
Norman Feske
41575ad60d sculpt: update menu when folding condition changed 
The storage dialog is folded when activating the runtime view (e.g., by
clicking on the Genode Logo). This should happen immediately as response
of the mouse click.
 2019-03-18 15:56:24 +01:00
Norman Feske
122c404883 menu view: cleanly separate update, layout phases 
This patch improves the separation of the update and layout phases to
avoid superfluous geometry animations of its child widgets. Prior this
patch, 'Widget::geometry' was called in both phases, potentially
triggering geometry animations with intermediate values at the update
phase.

Related to issue #3221
 2019-03-18 15:56:24 +01:00
Norman Feske
ba5de21db4 menu view: fade dependency changes 
This patch applies fading when the visibility of dependencies changes.

Issue #3221
 2019-03-18 15:56:24 +01:00
Norman Feske
b3219fc3a4 sculpt: fine-tune motion of graph positioning 
By using 30 motion steps, the positioning of the graph window fits
better with the geometry animations of the graph nodes inside the
window.
 2019-03-18 15:56:24 +01:00
Norman Feske
905da4b0cc menu_view: fade between button styles 
The button widget already supported an animated transition between
hovered and unhovered states. This patch generalizes the mechanism to
allow animated transitions between arbitrary button states, including
style changes.

This way, the fade-out of non-TCB components in Sculpt CE happens not
abruptly but smooth.

Fixes #3221
 2019-03-18 15:56:24 +01:00
Norman Feske
116bfab449 icon_painter.h: customizable pixel transfer 
This patch makes the application of color/alpha from the icon's texture
to the target surface customizable by replacing the formerly built-in
'_transfer_pixel' function by calls to the new 'Pixel_rgba::transfer'
interface.

Issue #3221
 2019-03-18 15:56:24 +01:00
Norman Feske
ebd9f36b0d os: Pixel_rgba::transfer interface 
The new 'transfer' function interface defines how pixel/alpha values
sampled from texture are applied to a destination pixel, similar to the
role of a fragment shader in GPU-based rendering. The transfer function
can be customized by defining custom pixel types, which may be (but
don't need to be) derived from 'Pixel_rgba'.

Issue #3221
 2019-03-18 15:56:24 +01:00
Norman Feske
b00e1f4e4b menu view: position widgets initially at (0, 0) 
The default 'Rect' constructor constructs an invalid rectangle where the
p1 coordinates are lower than the p2 coordinates. In particular, p1 is
set to (1, 1). The 'Widget' implementation uses the points individually
as input into the 'Animated_rect' mechanism. This way, widgets end up
being positioned at (1, 1) initially and are moved to (0, 0) once the
first layout update is applied. By explicitly initializing the
'_geometry' to (0x0+0+0), we avoid this initial artifact.
 2019-03-18 15:56:24 +01:00
Christian Prochaska
f4542b378c arora: update Nitpicker and shape report routes 
Fixes #3219
 2019-03-18 15:56:24 +01:00
Alexander Boettcher
0c6548fda0 sculpt: increase leitzentrale memory 
to avoid hanging popup and warning:

[leitzentrale -> gui] Warning: popup assigned RAM (12M) exceeds available RAM (6596441)

Issue #3218
 2019-03-18 15:56:24 +01:00
Norman Feske
669aed0ac0 sculpt: don't reset hover on click on '+' menu 
When entering/leaving sub menus of Sculpt's '+' menu, some parts of the
menu sometimes remain unchanged, in particular the back button.
Originally, a click would reset the hovering on clicks in the
expectation that any click would eventually result in a completely new
situation where the old hovering information does not make sense and
would only (potentially) confuse the menu. But this was apparently
overzealous. With the patch a once hovered back button stays hovered
even when actitivated and the back button of the upper-level menu
happens to stay under the current pointer position.

Issue #3209
 2019-03-18 15:56:23 +01:00
Norman Feske
014e800e8a menu view: immediate hover update on dialog change 
This patch improves the hover handling in situations where the dialog
changes under the pointer. Previously, hover changes were reported
as response to user input only, which failed to cover this case. This
became a problem with Sculpt CE's '+' menu, which changes on the fly
when entering/leaving sub menus.

The patch also cleanly separates the hover handling from the focus
handling. Originally, the hovering was reset when the menu view got
unfocused. In situations like Sculpt's '+' menu where the menu view
receives a transient focus only while clicked and gets unfocused on the
button-release event (aka clack), each clack would invalidate the hover
information until a new input event comes in.

Finally, the patch introduces the clear distinction between situations
where the entire dialog is hovered or not. Previously, this state was
somehow implicitly kept by issuing an invalid hover report whenever a
leave event was observed.

Issue #3209
 2019-03-18 15:56:23 +01:00
Norman Feske
a61c5b6be3 wm: move virtual pointer pos on window movement 
When a window is moved, the virtual pointer position must be updated,
taking the changed input coordinate into account. This patch propagates
such changes via absolute motion events to the client.

Without this patch, Sculpt CE's '+' menu wouldn't update the hovered
item correctly when entering/leaving sub menus (which happen to trigger
the repositioning of the menu on screen).

Issue #3209
 2019-03-18 15:56:23 +01:00
Norman Feske
ac3509d308 News item about Genodians.org 2019-03-18 15:56:23 +01:00
Norman Feske
30505c7977 test-spark: add object-size assertions 2019-03-18 15:56:23 +01:00
Christian Prochaska
2a71c8fa82 pcsc-lite: read vendor id and product id from USB device 
Fixes #3211
 2019-03-18 15:56:23 +01:00
Stefan Kalkowski
4796f761b8 base: initialize Cpu_state consistently (Fix #1452) 2019-03-18 15:56:23 +01:00
Christian Helmuth
18546ea16d Use https URLs for qt5 downloads 
The site download.qt.io seems to apply some mirror-via-HTTP-redirect
scheme which, unfortunately, seems broken (and therefore annoyingly
slow) for non-SSL-secured URLs.
 2019-03-18 15:56:23 +01:00
Josef Söntgen
0ef784b8fd mixer: only provide Audio_out in pkg 
The mixer only supports Audio_out.

Fixes #3210.
 2019-03-18 15:56:23 +01:00
Stefan Kalkowski
80fa23da5e hw: increase timing accuracy of kernel (fix #3081) 
* Introduce 64-bit tick counter
* Let the timer always count when possible, also if it already fired
* Simplify the kernel syscall API to have one current time call,
  which returns the elapsed microseconds since boot
 2019-03-18 15:56:23 +01:00
Stefan Kalkowski
2ecf1d887b hw: schedule on demand (Fix #3157) 2019-03-18 15:56:23 +01:00
Alexander Senier
a58fcc3b1e Update runtime ALIs 2019-03-18 15:56:23 +01:00
Alexander Senier
4c8d787918 Port libsparkcrypto 2019-03-18 15:56:23 +01:00
Emery Hemingway
127e5a2726 Add more features and symbols to libcrypto 
Fix #3203
 2019-03-18 15:56:23 +01:00
Johannes Schlatow
7bd506b913 virtualbox: fix compiler error with newer iasl versions 
I also added another sed command to suppress warnings that appear
because the include-guards contain dashes.

fixes #3147
 2019-03-18 15:56:23 +01:00
Norman Feske
bd09783438 sculpt: dim non-TCB when selecting a component 2019-03-18 15:56:22 +01:00
Norman Feske
ef0a3f5be1 menu_view: hidden dependencies in <depgraph> 
When specifying the attribute 'dep_visible="false"' for a primary
dependency or the attribute 'visible="false"' for a secondary
dependency, the dependency is used for the layout calculation but not
displayed in the graph.
 2019-03-18 15:56:22 +01:00
Norman Feske
653f653c67 menu_view: style for unimportant frame and button 2019-03-18 15:56:22 +01:00
Norman Feske
ac37991b34 sculpt: minor graph improvements 
- Omit showing routes to uninteresting ROMs obtained from the parent,
  i.e., the binaries requested by the sculpt-managed subsystems.
- Change the routes for the inspect subsystem such that the inspect-noux
  instance is anchored at the config node (critical!) and the nit_fb
  instance anchored at the used GUI.
 2019-03-18 15:56:22 +01:00
Norman Feske
6ee7049736 sculpt: add whitespace around parent resources 2019-03-18 15:56:22 +01:00
Emery Hemingway
57fd4e9148 Add Io_progress_handler to Entrypoint interface 
The "schedule_post_signal_hook" method of the Genode::Entrypoint class
is problematic because the signal hook can be scheduled and replaced
multiple times during the signal dispatch cycle. Add an alternative to
this method with "register_io_progress_handler" and the "Post_signal_
hook" class with "Io_progress_handler". The difference being an
"Io_progress_handler" may be registered once during the lifetime of an
entrypoint to prevent arbitrary libraries from replacing a pending hook.

The "register_io_progress_handler" remains as a deprecated API, and is
now invoked for every I/O signal received and only for I/O signals
rather than for any signal.

Ref #3132
 2019-03-18 15:56:22 +01:00
Emery Hemingway
93fb79f357 Check for existing directories at Vfs::Dir_file_system 
Return OPENDIR_ERR_NODE_ALREADY_EXISTS for existing directories.
Remove deprecated 'is_directory' method.

Fix #3083
 2019-03-04 10:57:04 +01:00
Norman Feske
c41fbe9ea5 sculpt: don't mark in-construction pkgs as present 
This patch excludes the current "Construction" from the list of
"present" components in the runtime. Without the patch, a missing "wm"
would go unmissing once when the routing dialog of a new wm instance
appears. Now an already present window layouter that had a broken route
would prematurely re-appear in the config, which should not happen
because the new wm does not exist yet.
 2019-03-04 10:53:37 +01:00
Alexander Boettcher
1cc3472e1d dde_ipxe: enable Intel i211 network card 
Fixes #3201
 2019-03-04 09:20:24 +01:00
Norman Feske
00ac29e53d Sculpt: avoid superfluous vertical space w/o info 
When a <pkg> node of an index lacks an 'info' attribute, omit the
corresponding fields in the pkg dialog to avoid blank vertical space.
 2019-03-01 19:27:40 +01:00
Norman Feske
434783da4d Sculpt: replace '_' by whitespace in GUI 
Fixes #3190
 2019-03-01 19:27:23 +01:00
Norman Feske
71019eccbd doc: minor tweaks of release notes 19.02 2019-03-01 16:45:24 +01:00
Norman Feske
06d68e87ba sculpt: list all depot users in selection menu 
Fixes #3188
 2019-03-01 16:45:24 +01:00
Christian Helmuth
8c2235c9a7 depot: delete signature file if signing failed 
In cases where the signing failed (maybe the passphrase for the key was
not available at this time), there remained an empty .sig file, which is
newer than the to-be-signed file and, therefore, prevents subsequent
signing processes.
 2019-03-01 15:43:50 +01:00
Norman Feske
cd244c2077 sculpt: don't install any index by default 
When buiding the sculpt image, the sculpt.run script used to integrate
the current version of the index of the 'depot_user' into the boot
image. At runtime, when the Sculpt partition is selected for "use" this
index - along with the 'pubkey' and 'download' files of the known
depot users - is written to the Sculpt partition.

This has the undesirable effect that a later version of the index
(published some time after the sculpt image was created) would always be
overwritten by the outdated index shipped with the boot image.

The built-in default index was actually a stop-gap solution needed
during the development of Sculpt CE, introduced when the downloading of
index files was not yet supported. Now, with the working download
mechanism, it is no longer needed. Hence, this patch removes the default
index from the sculpt image.
 2019-03-01 14:06:22 +01:00
Alexander Boettcher
30ddae0f91 nova: evaluate second kernel buddy earlier 
Fixes #3195
 2019-02-28 16:37:53 +01:00
Alexander Boettcher
38d3577c10 pkg: limit vbox5-nova-sculpt runtime to 4G 
Fixes #3197
 2019-02-28 16:36:12 +01:00
Emery Hemingway
bffcffe072 Skip Solo5 timer test for auto-QEMU 
Fix #3191
 2019-02-28 15:38:01 +01:00
6725 changed files with 334088 additions and 238208 deletions
Expand all files Collapse all files

6
LICENSE
View File

@@ -41,7 +41,7 @@
GNU AFFERO GENERAL PUBLIC LICENSE
Version 3, 19 November 2007
Copyright (C) 2007 Free Software Foundation, Inc. <http://fsf.org/>
Copyright (C) 2007 Free Software Foundation, Inc. <https://fsf.org/>
Everyone is permitted to copy and distribute verbatim copies
of this license document, but changing it is not allowed.
@@ -683,7 +683,7 @@ the "copyright" line and a pointer to where the full notice is found.
GNU Affero General Public License for more details.
You should have received a copy of the GNU Affero General Public License
along with this program. If not, see <http://www.gnu.org/licenses/>.
along with this program. If not, see <https://www.gnu.org/licenses/>.
Also add information on how to contact you by electronic and paper mail.
@@ -698,4 +698,4 @@ specific requirements.
You should also get your employer (if you work as a programmer) or school,
if any, to sign a "copyright disclaimer" for the program, if necessary.
For more information on this, and how to apply and follow the GNU AGPL, see
<http://www.gnu.org/licenses/>.
<https://www.gnu.org/licenses/>.

26
README
View File

@@ -65,34 +65,24 @@ The source tree is composed of the following subdirectories:
:'doc':
This directory contains general documentation. Please consider the following
document for a quick guide to get started with the framework:
! doc/getting_started.txt
If you are curious about the ready-to-use components that come with the
framework, please review the components overview:
! doc/components.txt
This directory contains general documentation along with a comprehensive
collection of release notes.
:'repos':
This directory contains the so-called source-code repositories of Genode.
Please refer to the README file in the 'repos' directory to learn more
about the roles of the individual repositories.
This directory contains the source code, organized in so-called source-code
repositories. Please refer to the README file in the 'repos' directory to
learn more about the roles of the individual repositories.
:'tool':
Source-code management tools and scripts. Please refer to the README file
contained in the directory.
:'depot' and 'public':
:'depot':
Local depot and public archive of Genode packages. Please refer to
! doc/depot.txt
for more details.
Directory used by Genode's package-management tools. It contains the public
keys and download locations of software providers.
Additional community-maintained components

2
VERSION
View File

@@ -1 +1 @@
19.02
21.08

54
depot/cnuke/pubkey
View File

@@ -12,19 +12,19 @@ sCaRvwPZdRCDDdhObkgkMlYROVNdC7ju8jZmB4n5O/5N7Ll7/RVhUWD7KeJu1UTM
oNEmhxEMrEcYcHFt8N8YVtJleRMVnsrZBNxOkFnpsPZr02XIQKfYi5tqSaBQZ47h
TTtXP3+FEaU+EoJprWqH55Sh97Fg6vuBJEmcGJeMGudFrypTzwqnM22DHwARAQAB
tCdKb3NlZiBTw7ZudGdlbiA8Y251a2VAZGVwb3QuZ2Vub2RlLm9yZz6JAlQEEwEI
AD4WIQSFq6G0496kOfRE2qnpSz5BlOa1+AUCWpAOsgIbAwUJA8JnAAULCQgHAgYV
CAkKCwIEFgIDAQIeAQIXgAAKCRDpSz5BlOa1+NGTEACg69FezAbc9Rzeb7j66NwW
3x/Zpi/jbmezMEtCZqAOcR8HJ7C0DN8gmCWPPB6oxAEeyL/i2cUb+9F0fTD6N7OL
TSOH75mNlyB9b8D2HdDILnLy4ClEitEOHFLMnlP58PGVtUNgbmsiM6cLLQtlJKvg
1xHlBTG9Ic8qgBcd808lLSC1XWN+nufVYBTE2RHNZqGeIWqPG5z1eW/JJO4M3V3j
MAw1p5Rpf9G6iWNxURutQOl/ii+97IKmHXX7N6ZRzawMGkOCAGNk1xeI71wCrALv
i3m7NmXb6XIJjD88hO5xfjjRO/0Gcw3vdmTXcVNzM1TvWj8ZsU6XjnMvNxa4LHqh
MxWuHlX0cAeCOzo1LJZ54f13hJ3YlTCrv68UIoNGRVi1LEbpE0sU0Ycdbw7ur2Gt
8Brx5WXv3hXTL8s3fmJzYU3/cFDfXjnaRiMWcNYdvckdHy6R7zPTk1NOvmfkolnO
W73ivdbonXpn0YqRGo2L76GdahLaVYn2gEeisqHnyeBCvI2snI+dMlEK0YNFrEkP
wgtWnF131Oorn0HMS6mpjcq4su4s5d84KCMGevFnW69oBRG27Bk/fJQiv2dSqkZg
+3TnyNApg1G7Bp5uD83mOvJWKdTWAntCJSQGbg1dmEsOBG+Bu7EE5LVBoMH3h/LQ
GB7c5NpgNIFQ2DZy2vkOI7kCDQRakA6yARAAws48PKN3BQPEM0M4kTO57/OqwGrA
AD4CGwMFCwkIBwIGFQgJCgsCBBYCAwECHgECF4AWIQSFq6G0496kOfRE2qnpSz5B
lOa1+AUCXk6dFAUJB4D1YgAKCRDpSz5BlOa1+JHXEACyB6vfzfuyo8fu1IaGrloC
qUUuSfhdleF1tPhZQn9W92bEEoXPfqFUuiNaxeFIXonyiD7wJYdcowdIb6CiNWrx
B8yjDQ7yb08sDLlL821I93kByy24Er5hY3QEMuEL3qv8YD/6Bh36Q0aoTXqQXMuQ
3AcHp0JI932Oez3q+gEdW7CUe40N9LaTx0A7ZTHe0i8IELkUPFtDg8xibOjZhBMR
vlxNUtzQVvC5kT5Wt0w6ZFKPGJguFylaE+TcsTNtiLNS4Z8PmRPgygrWotmprELo
2/py/VMxgQb7p4LFj+sNmU9x3ulFO3Efb9cVZVjBMYx5xh83ZiLZ3PvK43Eo+7nm
kYQhPinAjNmqng4/ZdQFrayFMkZc4oKCqV6BaYnybbsSn2ZEwVQqpDMeswWXTf9k
qC6DONz3ETtyzBEF8cHDpy+BwHRXx+pysGLhIvHS9C3M91j8oXvogZRbw/dEKGX2
eWes7nsdHeTbyRJunCCnsXDm0Js8cZcqj3Eu3Kpeu2CbgJGD+ajOel2RSMo2mFtg
OoRWjCibWHDSeUS+LGtspDpry+4VYwm4KBdcyEgiXnE8WDb/xk/hZgJCr2814AJC
RPpXoIZdFiwSP46vgoAnHynPbGTmE/Wt+F9Y2cn/dz6rmg5ynS8AKuTOUSoAKEUV
J20mBJpkSx7Cc95AGE8K/LkCDQRakA6yARAAws48PKN3BQPEM0M4kTO57/OqwGrA
AMlxnB3n+AP7rtZEC2L548+3pfg7Ub45Oq1X0ySxgPhV15CDUEK69pgs0JhphMbM
DyaUp04DyQWFhdueAqQVFnKvg6nZm/WIJF1VYEc0/q6Ramsdv2cWdxFBkFHYsH3P
G2bXCZZAEO6eJI8cdmBH6+bVptky/AybzBLztp1ruWlj+rm79qULmW7zJfxYwL1U
@@ -35,18 +35,18 @@ kAVN0Y0H9EGSEoVyKwS1DErzcBgajPKWedKlmjU1uIzupRWn5oqetVbNfZ3Z/EJ/
HdMCX0PN1kxg4WoI9mkP/moly1yopVOTTLJwvC2C85NQUmxyZeb4h9O7toFczBxn
7pV9Eprm/xRcO3ZEEpmdM7gR3+R3PpxkjgPIZAn9il32VYzdT3eQmDZ8sKC0qASG
ik2if34YnAggGmsrVB0nzT9fZR7CQq2IR7eohczLJRl+n7USQMgRweF+sl9PcbDT
aC0b0i+VveSEBOEAEQEAAYkCPAQYAQgAJhYhBIWrobTj3qQ59ETaqelLPkGU5rX4
BQJakA6yAhsMBQkDwmcAAAoJEOlLPkGU5rX4QjEQAIJF78XjvGomhafMsWlcd7fP
/j45B+KIpyruc7wHtLqWibi+YoDuvtG4m1/4Ckcz5qNrV0f5nQojEAeCSCh7Sl+s
yAJ9tmP1XET29DJq1t0iMsu+RDCLhdOfL/Wi/YJARtDloYbDlD8Rc/JnL2aOx2W/
Ybajj2lloxSDxKCnzCh1aZixie1YaQSm8ErshT2k4qTx48D3mRoBLAYyzdEbLkl3
ZBGDQy3Xk/miJ/hsj6L3w3G2YjMywZZZEjgDSUgSJ6MazBTgMmbCy1/0YGh7rNF7
iUoWyDZq4qiDGNAI695I6tas7s4X2dhZn10xgbJoa5Dq2rqQLek+ErEsr3DFYN53
7e9H0vJ9ydXB7piRJ9bxVFBng5CX9677IN5k9T/0lvcZWAFnDcHiMkIjIrTQ1Y04
WkG6DMVvzLD+PKA07cZLf692rsrbfpP4sGj8X6baWh58mGAMMwZ2Dy07grsi73qI
p/H4r4c/xRsH6nTbvb40zLfqcCz1xOTEcDtflYGMoeshu/H2EoruNMFi3c/9WO2G
3zrSOUenWFSiacbsh8HVE9HUFGddBTB3SrLddO5vvvi15OvGI4aWpnkW+9s4107h
jE0d0+c/Xm4AS0T59OGj/cd45k/vRe5vaPhQobZ6hXBzkKWPVt5uH8xbROiGiqdR
jMuaeWIldeJ60aeu39Dh
=JE0L
aC0b0i+VveSEBOEAEQEAAYkCPAQYAQgAJgIbDBYhBIWrobTj3qQ59ETaqelLPkGU
5rX4BQJeTp05BQkHgPWHAAoJEOlLPkGU5rX4RRAQAKSsr//mEXy6iQyoqaxJ796s
ljaoqsmT4ly/+0k9diX1+/wkulSkj4/fL2fDvRQK2cU1i6mYJVugN5swqXvp8WF8
scnDS/H75v/3Z+clTlsMjZ4qI0PmXYDpcvp5+zU85USEjOxpHmIKvQGflZ0NMvkI
uo2ZC6F4xnB5iizx9Ebm0gGAiKKEJ8ID/LLhJmOtidnGc3322SUVXIl4+ue9KgpO
hTBjx5UdIQ+21uSRf+zv3ZQH/KA/08iNyD20fIiUZKfEoeovlW8SipoO/pf/+1DE
WSiJqL3K7T3ixkC4oWSp7tbo2MWU247PItxTnnGF+KjwbkzpPwyPYPHT5Ed3XEZV
e39r9xUzdvYKKDCNDToDFsh4bm4hq7xXgnTwAbF6hw0iyBK02ZB6FKJsF/ZJi8gC
Ni4d3Lb54CV/Z7nAqMFbTa6KT8o9fc0Bw6tgX2ch7SRWU70yVGYmQzrdRsOtDkIV
iYA05JozwwsIcJQS7hBVnsqgD/huYLd+y7iCUgw35NdQ5764jzVlUpYtFO0NFrM6
QSKnnFFRiwr3+ZqIja2SGW455uP1hsl2ccA4Rdp/JRuEI6ojIo0R+JBpcMeqdv9d
HcNsAoZkrm/cQfIwlyKsU4/dEFL09kEtv7x/aO2ImqjAZ5xT3ml+4kFGEn7M7rsc
M6R6fgkYrVo1GhjXTlCL
=4bj4
-----END PGP PUBLIC KEY BLOCK-----

1
depot/ehmry/download
View File

@@ -1 +0,0 @@
https://depot.fuzzlabs.org

37
depot/ehmry/pubkey
View File

@@ -1,37 +0,0 @@
-----BEGIN PGP PUBLIC KEY BLOCK-----
Version: GnuPG v1
mQENBFm3BlYBCADR2ZGIWiYurf/urQ+rAZs3HMobXQ/fckc3FOAdY9qiklrvPr9s
maZZyMF+Wn4J2AmsGctJbBUt3ujcmk09AgZIzrHpVJRagUSz9q+bK7NvAfenDC2q
w4HS5yE9DYxXOjxrcS5a2HhKfE5sZef1zPtFIkEZCJXCvh4IL6maLijOrUZPf9R6
VJ+0itNh0hK9KiYsKA7CwrWuFaw7ZenVkEBV1HGXybFh9aInTTSqopyU6qYLDK1X
U0NG24cjM8TDbMGTaoScpLchcftBGEhjYg7Y2FCu455MDVdrmoOoxDINYgDU1KN6
8GBKKJ+xqG703/o550l+rCMB+TxMN5zKTFhXABEBAAG0IkVtZXJ5IEhlbWluZ3dh
eSA8ZWhtcnlAcG9zdGVvLm5ldD6JAVQEEwEIAD4CGwMFCwkIBwIGFQgJCgsCBBYC
AwECHgECF4AWIQRuEEFO5xYHgfX+6P+xTQQ6JrPSxwUCWgHubgUJAZm9OgAKCRCx
TQQ6JrPSxyJ0B/0eBOIloo3zJwtuQnNXe2ZFAmHFF7MJhrr0IpZ0n5K0OHTFPqXN
VJ+vhTGuI/SlF3a9YEHohIT2KKal1aq+/wVkX6Kdi4G4nGTEXdRsTwlzN/GXHBPF
IW0iUOInJOliKReaamsR/rgXXH5sstq2IU4x+NpHMT/OrkQPNSqVX0RjS9i76Awo
+EqdtW8g3y8qWd8WtTfhbMlQLhuzm99aADbspr4nK9WhZ7UHYLJAz/u7LzZlRiI5
hNa85BCPhNuYz5NTHhn4L9r6PdbKPzUV173XrlAtlteYuGIHk4d/zSAwlgDevChL
0MDRq1vvgfQ2V0SwIqRqIfnO31Ph8uktJ8WRiQEfBBABCAAJBQJaAReyAgcAAAoJ
EIZq4LBroLaERpQH/3Dw0H6N9YF2ATmkse6NNNQjxhKSUoA34UBWj0v4Q1AvXEe6
nHRR0mAa9Gije9ixKUg85VbKM3lUhXCFqW5jzyzpF1rzzTpAHaUTYKb8M2D9iRKo
9ejTK3KrgpstYJ8jthfosxeC+lK24WUXKKWPjgoIhv7Thd31ea/PZdkFVd1MpSwg
5BHRHcogsJI9uou7zWYbap9/k0lxKHJWklLh/oXet+aAoUIL0kTU+nILy3/RZJGW
9WlnZR9g3qSQ8YNpgNSrraEtXpcs6MGsWZSzbzJ1U95NL87NbWEOYgHZA++vhEy5
EXVrd8SuSHbUbqNF36gz1cxmEuKwxAaWZ1/UbGW5AQ0EWbcGVgEIANC9LozG/TXV
4j0TniuLrsq7kkNH+OJQexcD19pBrmEV4eYYD6c8BB9CaBITqvKIqtG0VJrmB5S5
D0oW4mQI+TkcX+Q7y3S7dUkc856d03aUX3vVcNhXEbNgM19qxAGieC8WH2kr4XU1
psu1WvFCXOktrE5c8H0rPCQ1jseB0S67XCA9BLwAqlqu0OuTpbWddjXuE1ahi0ig
Yt5YsHvgCMLup208azQqGCnJ5tFotqzbLg3I6+QzMcLySqsN4BzYmUGbOSSZdUhX
DWxwAK1dB8qa27iwsxCMncdGnqOTkq+mXRmu8ztPTUivyC6YHJ+O5lXvMMADZi4j
8RpQQlh5gFkAEQEAAYkBNgQYAQgAIBYhBG4QQU7nFgeB9f7o/7FNBDoms9LHBQJZ
twZWAhsMAAoJELFNBDoms9LHTmIH/0PuzpA3Ezdrb4xfVwnhUUZGiqfNzULCZipT
qtfym6Ove6To4BbXAihWwnCYs4sgipabW1WaXsDDvU+AmOmz4EPyTna23fUzYRSL
MT/kk77d+BGRGeA8/Gj/BgjPlHlFa7ukbT1DL1lSV0RMQ32tDGnhJ7wMBx4GoYU+
xjlsxrYKKdvZKOQVCCFn4KY/GavqT+8pYYI63stSPGspboQR2iboKqc350MVMvzz
IaDt14YiOYlgV6LoNj2VaPJvFqm9ML+plYl4PPQZTV6KAhFWCDJq2DzEQqwG6RU4
SsneHAqMlG4A5SSMTLEd9F2gZ2sJicybr1WbNLj1KF74ckUhpM4=
=p4q9
-----END PGP PUBLIC KEY BLOCK-----

1
depot/jschlatow/download Normal file
View File

@@ -0,0 +1 @@
https://depot.genode.org

59
depot/jschlatow/pubkey Normal file
View File

@@ -0,0 +1,59 @@
-----BEGIN PGP PUBLIC KEY BLOCK-----
mQINBGBlsG4BEADBMY44lF/LYiZByj1clJa+aNCEMGtoulxhFgCLIt2DR9r5KRmi
iAiweobH6ofbSXYKTOZqhAcmmkw3ihIVCsd1mHbdAAhlsnJetHLgOPL90aXVwce/
CM5QDubFmSu1VuaCgp/GaPqtQ/yaIq4LPMK+w2BfNG2M5XBeM8qLzjIHF7KNN4nZ
CX48Gx7q2DtIYCMUdIP0oK4dNfPDy6tvXiYvcz3lA2SNa6kszW3/mgIsurUEUFF6
vtCjBS5f6vJikWsglvTXqS0Sw5aIPzvosL/U0fjM0BVdi2NK6dWfM4HUduzr5MTD
Xc26CETmZ/8Sl4+mj0oUAuHU0SKDm22LNc+qRzXZxtkryDbFuikwtioYf3nIDTiA
63YPsx1jHp3mitTQgvRlJdlncmKIxXIUyGXqWpc+QHGUsvbDFdsXRxtt3J+8QWso
yLlaeP54E0OTQqdm+EnWGHahEKIwsn8/Gfs87phQsaF8Dlw+A132RukxB6zD35xF
/rqu25/8B0tL603Z3TLIyLDNbh4xO/iNohTB1ps+ALRwu48MsXcTMGUeWuRt2YVJ
RPG3KfIfnd+KcvJa2XFuzZCk57Jc8qGPmINPY5fJ7F9qIT0/9O7dlxzdoksfKXMf
RdiSCiFgmgJTqpXyfEcV7GUHCj9e0WdCSmDIH13bbuUpaTQXxO5d+XkzGwARAQAB
tDVKb2hhbm5lcyBTY2hsYXRvdyA8am9oYW5uZXMuc2NobGF0b3dAZ2Vub2RlLWxh
YnMuY29tPokCVAQTAQgAPhYhBBHurPU5I/pstoojMcciWBOKaz+1BQJgZbBuAhsD
BQkJZgGABQsJCAcCBhUKCQgLAgQWAgMBAh4BAheAAAoJEMciWBOKaz+1L28P/R+T
usXX/AdgaTRSY3hFDVnhV+adcyFdmTKWey0lMUMWRGsALd1EIJbqk4HAA81jW68j
9XG/kZgkhGKz+vOwOD40Y6G6f1GfKIEknzWok5RgH4p0BT5+ZhbqfX3/Z6z0CsOD
mXAo1At9JvGmP6OnpHbCwutEa/KAB2fgrjjgGMZcBlDzIsBugbT+shEAE5et0b0J
TrsClqoO71wknMkNp0ijfvCT4VtiCq/UBD5XqYRtzDQC6dMic/dNXKp3JmcsYf17
DUNM7RVdVx+v9onWiUre9poVn9YgA/MYwVb4hMyjJEc9R8yQhENSJtdhX2pkXVF6
l4ONbkdaw3xJqVO1ouPBI68Ddirwfwk6kgvpkvJ+iCVQ2lWSvPbtrhmTGpeerfQD
vbFwtUX1OZhZma5Gq8n5r6VM7N29hQalYaKPBS9AoL3YBO51NsUwde/EmZydlYl6
C7Ny+2iuPgKCoXS2zAxM2t9Ar9K0ZhhaiEbSZHoT596M/cWnn483edt15CQZO5w0
QNBv5nNKh0J/GmM45TtbhUya9Cpc9Ad2SVYSlNB3KQgC/n+1UQm+8UHOBq/5nuC/
Ww2It8g1G9pkGijG/f7flh1qE7KqqYec3cMLWj8hHRYSak3omSYlt4aOJEQPinjo
XUw+WBi/Mjrnd6UxB6mNs0VADOHDrCmFBBHcfWIAiQEzBBABCAAdFiEEwNwMDXup
zMtoYvj6v6ZIQUWP8uIFAmBlu6wACgkQv6ZIQUWP8uKNqwf+IjqK19dU8zATB50s
x4aEqTF6U5J2xO4xiBJlWv55zewjmmOsjJZYKBJd4UzTHqG/0CRUiDzUmsn3q1o+
GEwC+bB/LQMi+MaSG4w901Hz2cZ/qXonHfOtISKxmW2wDcOlG8Xsds0X5gfLgIZv
4EYK+Nh1HqcEj2/RHRzfT8/WkDSC7AzSd8nuwQL03ZkAW5woh8vX5A/Fgj77mJb+
APi6flqnTXxsVnRLHP8ErWQ8Akg07xV5fsOt1bFpY0aKgw2HJCoFyPDzVR09QILo
Th3jiHBdNTlbd+Hb1kQJfIyhIpwkNRKBxdbM0wZoKdVMl7xiDMMDHkYpvNyBQTkJ
nug/1LkCDQRgZbBuARAAzwJSYFegsVTJxm/6N9gN3vcfbZkvxttQYhNefw2eog7a
UZq8Eog4LUuhDmuVt0CWZh5YOz7u4XwjxOzNk6fw630AKiWiNkYtiLYwPyT1tPcz
I0XI6EcIYShyOfVZ+egPUELkjdC+SVwIQttQCPpEYnzWjbg22n+XeNRUfAI4EGKj
H5bqhKx6sR2cmbB4FPUO1kpYoqhEVdk1wdZI7n93SDPSLsoJSO+xBMPj6HqfXeab
gvr3dpqNg05G5g5JQBznILEJBHZBuCfy01j7WNRftdwWCGfWAh53uq6ug1vilodd
T1jvWmIbFE66rljH+sbtkZwBuEbEDnwZxNK/g72aQkog3AeDljG1a8GjazN6/upe
JD8LfoE8gjlj9Ncg9eYvqnrGR3nv566i7cq7UO10Dl5d6bedpEIo2htm90AI3M8p
nmtAZ18cbQodwerFvdQcegdejlpYJmI/3xaPZQqPH0puhDpT3oA53ObmjPb12ztH
Za06+klrr1MYb+x4IQYAOO8wwVRY/+YS+YU+6mxePyPMUDJYnIgOcTkg04/O55MF
KghOumAY3KTFoE7VlWXrBbhE/MW+Owp8zbtyFovJbmo8H718JHBQwlxzd1cS2Fvd
7g7r5eqWZNl+bMJQmgA0ZTAI0gD5PJU+GJEhiky3e29H2tBKqozX539HCxdOJycA
EQEAAYkCPAQYAQgAJhYhBBHurPU5I/pstoojMcciWBOKaz+1BQJgZbBuAhsMBQkJ
ZgGAAAoJEMciWBOKaz+1y94QAMD//6IqvIQEvsmDsv0hDy/ryPgYA7+2gKQCrvsq
rdhXYrBYGwgUPAeuYKgIFQIE+UqFZDwMbZ38DozyXvg4QYTEYQtPOio5WZGMHsNa
VYXel8qOKAbSECuP6GFiyX+NsSKYzs+iQLCzzLJS2/N8U1TMbgOwpqcVovpKOHvE
XPTUTwnUP25ubY32ElslRACZ/REJIm1Ftmloef50qnFggfCQdSsjevl74Q2+QOHk
Nb5w+a+vM6CmTGM1kqEATSJfBkxS7jCnGH4H9t44cEYqECpOYm4eUjxpM/9rxMw8
itrVs8preL28zzgKydVLO1JAX4WAw+X1k1Q9EGcRcXRnlxptZ71Lgnv4qCshPAXk
Sm5acLgdoOOSDDCSGFtjenyDjoHWfDk22CBwmEtMDtQ02m0+IzkfpNizqKGHBpKm
/hsu36teJilJWdAQOmDyrb0VsRweWM+rqClq3GS1+r791836WxQlKvx0/eO0gZOP
QjQxOjeCzZ/H8h7HBZ4bwRjQzHGgIbl7k6WPIwmvM4vLQfB44YEi/zDW5ivBcDEo
2VnHwAC2EPiVRwhRB/9+I8U09Ngrg2SnO8MkgmZpob1amblnhItM1US8bq/KDKld
4sWgWcZO21jEqnEaep4n/9vaxvLHfFa7AYY2EbB+Dfrns80oAqHG78qFyzflnLxx
kLJD
=7iZV
-----END PGP PUBLIC KEY BLOCK-----

1
depot/mstein/download Normal file
View File

@@ -0,0 +1 @@
https://depot.genode.org

51
depot/mstein/pubkey Normal file
View File

@@ -0,0 +1,51 @@
-----BEGIN PGP PUBLIC KEY BLOCK-----
mQINBFuhfGYBEADOj0EW/U2O3koeGIuNGDuBDSzkYjncoBrrPN1z69hgK349xiP7
ZwrPBA0kJ3ctLcRQcOEDF5XJzjByuNk3JV6MuKFqwy2i9dDTHkv5pv7Na/R3L2CM
k79weVahjPllt9nnbSMMtYH0eL187EhLW9owd0CY9wDj1lhLISHpyKvb9vwiUAwQ
i+xypGgU+Qvf8BmfTDdjfP0n+Iw31z+R3xiHhWmhrs9gID2XVtGZIy9NQQrijJYS
1Hp8hjgH9Fzmj0GkxUI+2mEqLeEp6wBmmo+7PRzc+iRYJNocyRxDbmdE0eyh3P93
QABzZiWTU+Feq0MYa1FK55uFCFHe6MSFLq33CGYMzQ9oFQHjuEX1KqJT/OoumPgN
iEb8Bt3XmqiI8GejafLKIW8gO8tYLABdrEpFelnyXy12IIOpOwb8GrxFUEu8+bDb
Fc0YkV7cZ4NH2Kq/6/Or30gCA2XUpqhDNSxA6jer35a3ZzvgXvcRkNR/oOt+qt/3
u9ew+e6Xjque0PRWzdavVlKdYWzSV8umVw5DGgQj/rSn+8spog/5w1lElyUa/5Yx
Z+1bvOIyXPg+tTLimWtGRZQorak1sMsLMDqQck9SwOQWpSmVGsJ14uWXSMaZjdj9
4rHSSymYlcQWP8t9lfPf6/ZSH8574PbbJSe71UXcWWznSGprSt2QQAQ7NwARAQAB
tCtNYXJ0aW4gU3RlaW4gPG1hcnRpbi5zdGVpbkBnZW5vZGUtbGFicy5jb20+iQI+
BBMBAgAoBQJboXxmAhsDBQkDwmcABgsJCAcDAgYVCAIJCgsEFgIDAQIeAQIXgAAK
CRDJZVRAR8NPkiTSD/9xiUVNgKTkmH9WRdCcaHuyhKoQI6CsWnMOW5OZ+gVXPc9U
LapQ5Wymb+yGE8Uyn4EJnwdEa8TdWztMs3TNtK5vh1Pk1lKmtKcblBiDrUPEURhR
mUBoUkMe2VDXyN27e0lkbxcFJ0HrYnhvb72Yfnw6B3XMa/NIhbOU3ttB/tVR6Kjl
YVDtTokfILQc07Ai+TqkqNARtPGntHAn9qFvPLNKI31c+Xw/QGZbVPNFW1QAYBiO
5dqsSwKaouapHAb0GBmU0fAfJZNcvwt9Ph7WDXU9Ce3LZmIshGW7Gzm7EGdf4VS6
k4OtEWZhZHRnl9/sBJ2UlQU0tFs3ow53aDZQIa4FKd4JKabBeRgDUKCGEyfoNKCq
NKkLZaR/kFA9EqTPZSIXPLYIzIlSFoL3LVDOc5O7hDT9vXaubdZeAfE9VcHDdm3u
usW034F0zbsuu4G83qZ6QhDOp0UhujfSU+BDkH0I+OfFyVkQAmy69J3byi7Iluno
OcZM13rbe30jzyyyWAnjHicAt8tNwCLMmnIRvYT9ELqywID0Zzxkct61eJQ6L6aq
/DnUD9Kq0OwdxC1wMGtimX5eUnN7S7y5+TMRepJnWcTtgrLBVqwRrvSLnsKVd4Gb
Bu3dSs+BxMV1UcyU0KWwDPY2D7k8SGgeH7IEPo/JiGO8RrWYJRac9qel9egMYbkC
DQRboXxmARAAoaRH/xgKvp0QrNFJZBW12SdXT4h1nXglbpZd+bUl738LqM2JSIC5
Qq7t2DUtOiL9UxqvRgAIjkVLR+hHJBZyuPOP5Ebaf3Mr7eAdmCx2TMsHZPKBLnTy
EqeUcnPwThNKcUaGsQYlkxIQAJtPzy3YBuDCYnzzSi6KzP3VC8szHsXcPo2DeKWI
m9BbMMSQkB3m0fz8W3ar/64RDoubRWRJ07b7q4WUb4jPvql+Tz+Gv5x3qm8g6m1j
uD+U5OhGCe3m5/MsI2JYdWXEwif8YD59gylsj3LJIu3i4SkR8aS+x6n3LIvdSCOt
P3To87wfoSqBWWObVe/vOHRXqpzEF6T/ih/rkhUFzrHNMziTPsujBazm82AUZ6xi
2TddbdSAK+ZpSPQjyyRflNyNgv0iu6P3BnBgv0Xcuf2RsDMBDjdvHtX0lXkPMB+r
6jgxl0faV6UktH5R2d4Zt9i0/vWlZZd0MDylPNQDiKs+qSIX3ntfUUaPdEJqYAwM
gloXr3cZtsVh+Yoqt8pa4x0x6kqez+oyBUFf6y1FCKYq2H/QfPRkF/DrfZS+7jXW
iAX8PcrDceCN6Dr10/g5LPWXQOHUWxCu7FEYoYJsVXst4N0FnCOCe/OGYIhMteSV
IJJvNS4qwXZKKZ+cmuT9skEQ6zsLV9+VDYMwVqXB4wTp4EwT2pCp6LEAEQEAAYkC
JQQYAQIADwUCW6F8ZgIbDAUJA8JnAAAKCRDJZVRAR8NPktRiD/9qjH6uuT1y5uNA
4E8u6/vxZGsyBG57dtnQ0KQbcsZSrUxMiDTtMMrklsl2vtut/XDroajm+d2eaQ4K
j0h285VRkT5xJ79JaIQ8oyzRWBtpENy0iLUm7CmqCMuH8VFifZyEBYZgR/K+9ago
WwF3sk0QP0WJTFqIZajKK0Lo5Juk8DCXjvtkvGNytHkZ2vEkiZ9m7Eq0AjQnts5T
PdHkob7Z8fu0KfUMoCUtb+TzNyMYOVGE6uTfo8wKvbhwzUWSymZkrecx73XUUmew
MdZ2WzwGdUMXwcBZ4LvgrZVg+LKxR1lyJ+qbIUpDKXCZGGH2VZVaQspgnx2Yt0EI
Zl2ELMrTq5IcOkxtblEZpQkwaXszqUDNO43ezp/PmbTRWChoM/k3RAbp9rvoYx93
UChyRXpJRaqDxK502k3BSnOfmolpQvxTMKtLZO50URa0F0F4qD2csns4QrqOWsGr
qPqVKcLn8Q/yMW9fm5+xX+ZHICKk64GAB0YyHcw9lOWekLhGzZV1KnD5ID6+5E0B
nsWZB6PDqLM+/ykXh0rcN8DWwZvhimI5/dOEIWIkADKT8WGwh+2JnRyimhxYdaiH
ul1BvZAYqF1FVXAHd0u48MssLywGGbW4fH8D9SM+LFpP4+8oYFLNgH1OreMENuag
i47EaxpnQUgIt8PZ/gL3LeiGs8SNpw==
=flyM
-----END PGP PUBLIC KEY BLOCK-----

34
depot/skalk/pubkey
View File

@@ -6,26 +6,26 @@ spm0DdNCEVfUbIQUstbSNt1kDGIlqPvZmoRj5YwDECW6ULG8jTWhaAz9SybJCE86
jLPzZHjKtVhCVO1X0ogSxvvYGemUpqVCLfKcIb1TucieKKCrnjiHwp0XZ41DWqSb
kBwWW2YQMEsw8JNGNTfCCUFf6+1l4mMixtv2sEqFiH1wQJ0c4gSa4iSYZ+eNmzJ0
B13K8kONOctSg3NYZz/dC1aUzLOTkgJHqEPHABEBAAG0KVN0ZWZhbiBLYWxrb3dz
a2kgPHNrYWxrQGRlcG90Lmdlbm9kZS5vcmc+iQFUBBMBCAA+FiEEI2X1/oP+p+Ls
2L28qrC4CLnp/P8FAlrQZGACGwMFCQPCZwAFCwkIBwIGFQgJCgsCBBYCAwECHgEC
F4AACgkQqrC4CLnp/P9kKQgAxc11eDhYkMVg9cuipFoqtV5lY9aT2qkocZ28IhbF
LXhy9lcn5FlxZSVdkzJQ5tUf4nSuhhMb6z3r8edaOebcYAyFk0DTymNpcEyT2XHd
lsOcInhfU423m1tCNHdmxtv9HERtj2zS1KNkjWxY6xsqfEw1eDfUvdS3K9KpUqc6
vWZwsPd9EHxW3mzWJS3lrSAnNsCwtdmiqB9045Yss4KednMcN6qxE+uHppQ+25Ib
5ZICpiVqOJ+eQXeY3kRx84lfZJr3uFdn00RSU5fn0uol8ZZYX9tQd9SC1GIlkyYj
HxNLKNVaYzF1nnmR1s7cpY6PUAYbt0im6kd4VJ1wQcWTnrkBDQRaU+QuAQgAuame
a2kgPHNrYWxrQGRlcG90Lmdlbm9kZS5vcmc+iQFUBBMBCAA+AhsDBQsJCAcCBhUI
CQoLAgQWAgMBAh4BAheAFiEEI2X1/oP+p+Ls2L28qrC4CLnp/P8FAl4XHPUFCQ0p
OkcACgkQqrC4CLnp/P8bAgf/ToUKzXMGdRXpB+xIBbOxuu8xVdLRj+5cm7cjZRiy
N0ulrzAlaZqrfqb5rTujPDjakmVSBVRIi8zWlzU7y67fevTFdi/Q9xeLgbqP3cWU
UgUJs03cdacFIamSIO//I9gaImZ2kpUcA6Nj+N+jPepRgUBzljsJG8678ndicA0e
Yio4uAvZY9JmsWGn48rmkqshPf9lag4GUnXn+R/AFKCts55kOiPekux3pJBhmrLJ
TdLgdAzlzGQfHxQbz3oVK4yE19ReAd6rnpW06oK4KfiWrgCmaEk1quirDNfIMDfr
nwe6571TY5is0f9zJjJgP8ogLUMI3n54OvwMezUMz+bLbbkBDQRaU+QuAQgAuame
2VDCgNmiwu+QmWyNN4jzbE7+VNmDr37HO9lZRIROC4eACPOGfUL03jLGvUn7rrxQ
JK+Pit6gXXCoIWDhCMNRSZKho416KJZWxF2jxKBKGQ7DtWaTR3YOzSf0ka9DZSrp
wG22xS0Uf1U/0ZBIf24LbyUDFLc8zt4eey2D9AHm+9vCf7wnf8TV6SNIXRz3wj7d
VqoZLXRTT7twBSaNahLhNtg7fhS8Nu6/THuwXpNKPAvAsgJRTGk7kmrKj5P7rOZA
rNHC1pvK1EWJJi2onHOmCzBccKRp9SeQlj+ddqG6seZhEnRYG6l7uhkyNZoyvKxO
Fguc5g8Xf37VyuRMfwARAQABiQE8BBgBCAAmFiEEI2X1/oP+p+Ls2L28qrC4CLnp
/P8FAlpT5C4CGwwFCQPCZwAACgkQqrC4CLnp/P/IOAf+LnQVtU7aHh4AZDsi1wXq
KBo5l6r3G8tC/S0HEf8nnWMUio2/mwVrkbuTvBeKrcQ/mXFHHAG8YCAIHPgR7T0y
2L6l2PL4HoXiLD8EwJ0sWZu2waxuxcTX+bb1i3Xm3squnqDtCX3pyoXWx0GVgrz9
7I/zitxeER+35ScaZ+JAAcNW59LpiV1SdIXqbtrw5QBJBuZUp0bvnzCNvdZLhnhb
gWfwPEfcXFt5K87iTmfMFJOJpbUrEz/NWE9gOBCBjqxW0wVb+IWr0oFWvfxjuBq7
IW9DezwkN1wAavP6g7+B4esCD6SRq+3CCzbT1By3X2h3SevU8tHkCSA3cIKgWdyD
2A==
=IsaH
Fguc5g8Xf37VyuRMfwARAQABiQE8BBgBCAAmAhsMFiEEI2X1/oP+p+Ls2L28qrC4
CLnp/P8FAl4XHW4FCQ0pOsAACgkQqrC4CLnp/P/xGggAhMz6NH1dYiFtc9+3vkAW
bZ10MDvxdgyU+G1a0BVYVo2ZXkhJlb2yg5MEeL1MZ6cUtM7MR1OL0U1jayfdcTfr
kkUi2a35BdvKRpxc3hcc4i2CdDxdx+IGrBa75P+vIKFj8WIvwABgRJNVzY1Oq6Lq
zKPUpVPug80Hch++Kv6l1DEpcoXlGaI8Z1dWfAjYQ7HJF/YJNHloKwxa1mXo/l/S
qQSFHB23IN682ANm8iEUphuvJVbOw4mubpCEpmJvIhEBeesgXfB503nniOsshq+9
1bNZ3CFv/ylnj5sy5VzU0lfW/TrHiW4OOOSbX8mCSvcqN2mhjcu6r4olV0fWPrrV
Lg==
=H+8g
-----END PGP PUBLIC KEY BLOCK-----

14
doc/build_system.txt
View File

@@ -362,6 +362,20 @@ in its 'LIBS' declaration and refers to the tools relative to
'$(BUILD_BASE_DIR)'.
Building additional custom targets accompanying library or program
==================================================================
There are cases when it is important to build additional targets
besides standard files built for library or program. Of course there
is no problem with writing specific make rules for commands that
generate those target files but for them to be built a proper
dependency must be specified. To achieve it those additional targets
should be added to 'CUSTOM_TARGET_DEPS' variable like e.g. in
iwl_firmware library from dde_linux repository:
! CUSTOM_TARGET_DEPS += $(addprefix $(BIN_DIR)/,$(IMAGES))
Automated integration and testing
#################################

253
doc/challenges.txt
View File

@@ -16,17 +16,6 @@ research projects on Genode.
Applications and library infrastructure
#######################################
:GNU Privacy Guard:
The [https://gnupg.org/ - GNU Privacy Guard] (GNUPG) is the most widely
used Free-Software implementation of the OpenGPG standard. It comprises a
rich set of tools for encryption and key management. For many forthcoming
application scenarios of Genode such as package management and email
communication, GNUPG is crucial. Hence, it should be ported to Genode. Such
a port may leverage Genode's fine-grained component architecture to strongly
separate network-exposed functionality, the storage of key material, and the
cryptographic functions.
:VNC server implementing Genode's framebuffer session interface:
With 'Input' and 'Framebuffer', Genode provides two low-level interfaces
@@ -50,24 +39,6 @@ Applications and library infrastructure
integrated in the operating system, i.e., in the form of Genode components
or a set of Genode VFS plugins.
:Tiled window manager:
At Genode Labs, we pursue the goal to shape Genode into an general-purpose
operating system suitable for productive work. The feature set needed to
achieve this goal largely depends on the tools and applications daily used by
the Genode engineers. As one particularly important tool for being highly
productive, we identified a tiled user interface. Currently, all developers
at Genode Labs embrace either the Ion3 window manager or the tiled Terminator
terminal emulator. Hence, we desire to have a similar mode of user
interaction on Genode as well. The goal of this challenge is to identify the
most important usage patters and the implementation of a tiled GUI that
multiplexes the framebuffer into a set of tiled and tabbed virtual
framebuffers.
Related to this work, the low-level 'Framebuffer' and 'Input' interfaces
should be subject to a revision, for example for enabling the flexible change
of framebuffer sizes as needed by a tiled user interface.
:Interactive sound switchbox based on Genode's Audio_out session interface:
Since version 10.05, Genode features a highly flexible configuration concept
@@ -116,16 +87,33 @@ Applications and library infrastructure
of communicating threads as captured on the running system. The tool should
work on a selected kernel that provides a facility for tracing IPC messages.
The underlying light-weight tracing infrastructure is
[https://genode.org/documentation/release-notes/19.08#Tracinghttps://genode.org/documentation/release-notes/19.08#Tracing - already in place].
The Qt-based tracing tools would complement this infrastructure with
an interactive front end.
:Ports of popular software:
Genode features a ports mechanism to cleanly integrate 3rd-party software.
Thanks to the C runtime, the flexible per-component VFS, the standard
C++ library, and the Noux runtime (for UNIX software), porting software
to Genode is relatively straight forward. The
[http://genode.org/documentation/developer-resources/porting - porting guide]
[https://genode.org/documentation/developer-resources/porting - porting guide]
explains the typical steps. A wish list of software that we'd like to
have available on Genode is available at
[http://usr.sysret.de/jws/genode/porting_wishlist.html].
[https://usr.sysret.de/jws/genode/porting_wishlist.html].
:Native Open-Street-Maps (OSM) client:
When using Sculpt OS, we regularly need to spawn a fully fledged web
browser in a virtual machine for using OSM or Google maps. The goal
of this project would be a native component that makes maps functionality
directly available on Genode, alleviating the urge to reach for a SaaS
product. The work would include a review of existing OSM clients regarding
their feature sets and the feasibility of porting them to Genode.
Depending on the outcome of this review, an existing application could
be ported or a new component could be developed, e.g., leveraging Genode's
Qt support.
Application frameworks and runtime environments
@@ -133,20 +121,20 @@ Application frameworks and runtime environments
:OpenJDK:
[http://openjdk.java.net/ - OpenJDK] is the reference implementation of the
[https://openjdk.java.net/ - OpenJDK] is the reference implementation of the
Java programming language and hosts an enormous ecosystem of application
software. The goal of this line of work is the ability to run this
software directly on Genode. The centerpiece of OpenJDK is Hotspot - the
Java virtual machine implementation, which must be ported to Genode.
The initial port should suffice to execute simple example programs that
operate on textual input. Since Genode has the FreeBSD libc readily
available, OpenJDK's existing POSIX backends can be reused. The next step
is the creation of Genode-specific native classes that bridge the gap
between the Java world and Genode, in particular the glue code to
run graphical applications as clients of Genode's GUI server. Since
OpenJDK has been ported to numerous platforms (such as Haiku), there
exists a comforting number of implementations that can be taken as
reference.
software.
Since
[https://genode.org/documentation/release-notes/19.02#Showcase_of_a_Java-based_network_appliance - version 19.02],
Genode features a port of OpenJDK that allows the use of Java for networking
applications.
The next step would be the creation of Genode-specific native classes that
bridge the gap between the Java world and Genode, in particular the glue
code to run graphical applications as clients of Genode's GUI server. Since
OpenJDK has been ported to numerous platforms (such as Haiku), there exists
a comforting number of implementations that can be taken as reference.
:Android's ART VM natively on Genode:
@@ -155,22 +143,6 @@ Application frameworks and runtime environments
removed from the trusted computing base of Android, facilitating the use of
this mobile OS in high-assurance settings.
:Rust bindings for the Genode API:
Rust is a low-level systems programming language that ensures memory
safety without employing a garbage collector. It thereby challenges C++
as the go-to programming language for high-performance and low-level code.
Since
[http://genode.org/documentation/release-notes/16.05#New_support_for_the_Rust_programming_language - version 16.05],
Genode supports the use of the Rust programming language within
components. However, to unleash the potential of this combination,
Genode's API must become available to native Rust code. The intermediate goal
of this project is the implementation of an example server, e.g., a
component that provides a terminal-session interface. Thereby, we
will encounter the problems of bootstrapping and configuration of the
component, the provisioning of signal handlers and session objects, and
memory management.
:Go language runtime:
Go is a popular language in particular for web applications. In the past,
@@ -218,10 +190,37 @@ Application frameworks and runtime environments
analyzing such components with formal methods.
The use of Haskell for systems development was pioneered by the
[http://programatica.cs.pdx.edu/House/ - House Project]. A more recent
development is [http://halvm.org - HalVM] - a light-weight OS runtime for
[https://programatica.cs.pdx.edu/House/ - House Project]. A more recent
development is [https://halvm.org - HalVM] - a light-weight OS runtime for
Xen that is based on Haskell.
:Xlib compatibility:
Developments like Wayland notwithstanding, most application software on
GNU/Linux systems is built on top of the Xlib programming interface.
However, only a few parts of this wide interface are actually used today.
I.e., modern applications generally deal with pixel buffers instead of
relying on graphical drawing primitives of the X protocol. Hence, it seems
feasible to reimplement the most important parts of the Xlib interface to
target Genode's native GUI interfaces (nitpicker) directly. This would
allow us to port popular application software to Sculpt OS without
changing the application code.
:Bump-in-the-wire components for visualizing session interfaces:
Genode's session interfaces bear the potential for monitoring and
visualizing their use by plugging a graphical application
in-between any two components. For example, by intercepting block
requests issued by a block-session client to a block-device driver,
such a bump-in-the-wire component could visualize
the access patterns of a block device. Similar ideas could be pursued for
other session interfaces, like the audio-out (sound visualization) or NIC
session (live visualization of network communication).
The visualization of system behavior would offer valuable insights,
e.g., new opportunities for optimization. But more importantly, they
would be extremely fun to play with.
Virtualization
##############
@@ -237,21 +236,6 @@ Virtualization
is normally not possible. Also, complex Genode scenarios (like Turmvilla)
could be prototyped on GNU/Linux.
:VirtualBox on top of seL4:
The [https://sel4.systems - seL4 microkernel] is a modern microkernel that
undergoes formal verification to prove the absence of bugs. Since version
4.0, the kernel supports virtualization support on x86-based hardware.
Genode has experimental support for seL4 that allows almost all Genode
components to be used on top of this kernel. VirtualBox is an exception
because it closely interacts with the underlying kernel (like NOVA) to
attain good performance. We have shown that VirtualBox can be executed
within a protection domain of the NOVA microhypervisor. The goal of this
project is the application of this approach to the virtualization
interface of seL4. The result will be a VM hosting environment that
ensures the separation of virtual machines via the formally verified
seL4 kernel.
:Xen as kernel for Genode:
Using Xen as kernel for Genode would clear the way to remove the
@@ -294,22 +278,25 @@ Virtualization
the project bears the opportunity to explore the provisioning of the
KVM interface based on Genode's VFS plugin concept.
:Hardware-accelerated graphics for virtual machines:
In
[https://genode.org/documentation/release-notes/17.08#Hardware-accelerated_graphics_for_Intel_Gen-8_GPUs - Genode 17.08],
we introduced a GPU multiplexer for Intel Broadwell along with support
for Mesa-based 3D-accelerated applications.
While designing Genode's GPU-session interface, we also aimed at supporting
the hardware-accelerated graphics for Genode's virtual machine monitors like
VirtualBox or Seoul, but until now, we did not took the practical steps of
implementing a virtual GPU device model.
The goal of this project is the offering of a virtual GPU to a Linux guest
OS running on top of Genode's existing virtualization and driver
infrastructure.
Device drivers
##############
:Isochronous USB devices:
Genode's USB driver supports bulk and interrupt endpoints. Thereby, most
USB devices like USB storage, user input, printers, and networking devices
can be used. However, multi-media devices such as cameras or audio
equipment use isochronous endpoints, which are not supported. The goal
of this line of work is the support of these devices in Genode. The topic
touches the USB driver, the USB session interface, an example implementation
of a USB client driver (using the session interface) for a device of choice,
and - potentially - the enhancement of Genode's USB-pass-through mechanism
for VirtualBox.
:Sound on the Raspberry Pi:
The goal of this project is a component that uses the Raspberry Pi's
@@ -318,24 +305,12 @@ Device drivers
backend, the new driver will make the sound of all SDL-based games
available on the Raspberry Pi.
:Framebuffer for UEFI and Coreboot:
By moving away from the legacy BIOS boot mechanism, it is time to
reconsider closely related traditional approaches such as the use of
the VESA BIOS extensions for accessing the frame buffer. On UEFI or
Coreboot systems, there exist alternative ways to initialize and
access the framebuffer in a hardware-independent way. On the course of
this project, we will explore the available options and create dedicated
Genode driver components that use the modern mechanisms.
For reference, the current state of Genode's UEFI support is documented
in [https://github.com/genodelabs/genode/issues/2242 - Issue 2242].
:Data Plane Development Kit (DPDK):
Genode utilizes the network device drivers of the iPXE project, which
perform reasonably well for everyday use cases but are obviously not
designated for high-performance networking.
The [http://dpdk.org/ - DPDK] is a vendor-supported suite of network device
The [https://dpdk.org/ - DPDK] is a vendor-supported suite of network device
drivers that is specifically developed for high-performance applications.
It presents an attractive alternative to iPXE-based drivers. This project
has the goal to make DPDK drivers available as a Genode component.
@@ -357,8 +332,22 @@ Platforms
Genode functionality such as its native GUI, lwIP, and Noux, many protocol
stacks can effectively be removed from the Linux kernel.
The goal of this project is to evaluate how small the Linux kernel can get
when used as a microkernel.
In 2018, Johannes Kliemann pursued this topic to a state where Genode
could be used as init process atop a customized Linux kernel.
[https://lists.genode.org/pipermail/users/2018-May/006066.html - His work]
included the execution of Genode's regular device drivers for VESA and
PS/2 as regular Genode components so that Genode's interactive demo
scenario ran happily on a laptop. At this time, however, only parts of
his results were merged into Genode's mainline.
The goal of this project is to follow up on Johannes' work, bring the
[https://github.com/genodelabs/genode/pull/2829 - remaining parts] into
shape for the inclusion into Genode, and address outstanding topics, in
particular the handling of DMA by user-level device drivers. Further down
the road, it would be tempting to explore the use of
[https://en.wikipedia.org/wiki/Seccomp - seccomp] as sandboxing mechanism
for Genode on Linux and the improvement of the Linux-specific implementation
of Genode's object-capability model.
:Support for the HelenOS/SPARTAN kernel:
@@ -381,34 +370,46 @@ Platforms
kernel is used for Mac OS X, it could represent an industry-strength
base platform for Genode supporting all CPU features as used by Mac OS X.
:Linux process containers for supporting Genode`s resource trading:
:Genode on the Librem5 phone hardware:
Even though the Linux version of Genode is primarily meant as a development
platform, there exist interesting opportunities to explore when combining
Genode with Linux, in particular Linux' process containers.
Linux process containers provide a mechanism to partition physical resources,
foremost CPU time, between Linux processes. This raises the interesting
question of whether this mechanism could be used for a proper implementation
of Genode's resource trading on Linux.
[http://lwn.net/Articles/236038/ - Process containers introduction...]
Even though there exists a great variety of ARM-based SoCs, Genode
primarily focuses on the NXP i.MX family because it is - in contrast
to most SoCs in the consumer space - very liberal in terms of
good-quality public documentation and reference code, and it scales
from industrial to end-user-facing use cases (multi-media).
The [https://puri.sm/products/librem-5/ - Librem5] project - with its
mission to build a trustworthy mobile phone - has chosen the i.MX family as
the basis for their product for likely the same reasons that attract us.
To goal of this work is bringing Genode to the Librem5 hardware.
For the Librem5 project, Genode could pave the ground towards new use cases
like high-security markets where a regular Linux-based OS would not be
accepted. For the Genode community, the Librem5 hardware could become an
attractive mobile platform for everyday use, similar to how we developers
use our Genode-based [https://genode.org/download/sculpt - Sculpt OS] on our
laptops.
System management
#################
:Remote management of Sculpt OS via Puppet:
[https://en.wikipedia.org/wiki/Puppet_(company)#Puppet - Puppet] is a
software-configuration management tool for administering a large amount
of machines from one central place. Genode's
[https://genode.org/download/sculpt - Sculpt OS] lends itself to such
an approach of remote configuration management by the means of the
"config" file system (for configuring components and deployments) and
the "report" file system (for obtaining the runtime state of components).
The project would explore the application of the Puppet approach and tools
to Sculpt OS.
Optimizations
#############
:Low-latency audio streaming:
Genode comes with an audio streaming interface called 'Audio_out' session.
It is based on a shared-memory packet stream accompanied with asynchronous
data-flow signals. For real-time audio processing involving chains of Genode
components, streams of audio data must be carried at low latency, imposing
constraints to buffer sizes and the modes of operation of the audio mixer and
audio drivers. The goal of this project is to create a holistic design of the
whole chain of audio processing, taking thread-scheduling into account. A
particular challenge is the mixed output of real-time (small buffer, low
latency) and non-real-time (larger buffer to compensate jitter, higher
latency) audio sources.
:De-privileging the VESA graphics driver:
The VESA graphics driver executes the graphics initialization code provided

18
doc/coding_style.txt
View File

@@ -93,6 +93,24 @@ and the source code always looks good.
_Hint:_ In VIM, use the 'set list' and 'set listchars' commands to make tabs
and spaces visible.
* If class initializers span multiple lines, put the colon on a separate
line and indent the initializers using one tab. For example:
! Complicated_machinery(Material &material, Deadline deadline)
! :
! <tab>_material(material),
! <tab>_deadline(deadline),
! <tab>...
! {
! ...
! }
* Preferably place statements that alter the control flow - such as
'break', 'continue', or 'return' - at the beginning of a separate line,
followed by vertical space (a blank line or the closing brace of the
surrounding scope).
! if (early_return_possible)
! return;
Switch statements
~~~~~~~~~~~~~~~~~

500
doc/components.txt
View File

@@ -14,6 +14,11 @@ Genode comes with a growing number of components apparently scattered across
various repositories. This document provides an overview of these components
and outlines the systematics behind them.
The scope of this document is limited to the Genode main repository maintained
by Genode Labs. Many additional components and device drivers can be found in
the community-maintained
[https://github.com/genodelabs/genode-world/ - Genode-World] repository.
Categorization of components
############################
@@ -26,11 +31,11 @@ of them is briefly characterized as follows:
session interfaces. Naturally, a device driver is specific to a
particular hardware platform. The hardware resources are accessed
via core's IO_MEM, IO_PORT, and IRQ services. The functionality of
the driver is made available to other system components by announcing
the driver is made available to other system components via
one of Genode's device-independent session interfaces, which are
'platform_session', 'framebuffer_session', 'input_session', 'block_session',
'platform_session', 'capture_session', 'event_session', 'block_session',
'audio_out_session', 'log_session', 'nic_session', and 'timer_session'
(see 'os/include/' for the interface definitions). Those interfaces are
(see _os/include/_ for the interface definitions). Those interfaces are
uniform across hardware platforms and kernel base platforms. Usually,
each device driver can accommodate only one client at a time.
@@ -59,7 +64,7 @@ of them is briefly characterized as follows:
Device drivers
##############
Device drivers usually reside in the 'src/drivers' subdirectory of source-code
Device drivers usually reside in the _src/drivers/_ subdirectory of source-code
repositories. The most predominant repositories hosting device drivers are
'os', 'dde_ipxe', 'dde_linux'.
@@ -67,19 +72,23 @@ repositories. The most predominant repositories hosting device drivers are
Platform devices
================
:'os/src/drivers/platform/': Platform drivers for various platforms.
:_os/src/drivers/platform/_: Platform drivers for various platforms.
On x86, the platform driver uses the PCI controller as found on x86 PC
hardware. A client can probe for a particular device and request information
about physical device resources (using the 'platform_device' interface). I/O
resources for MMIO regions, I/O ports, and interrupts can be requested by the
provided device abstraction.
:'os/src/drivers/acpi':
:_os/src/drivers/acpi/_:
On x86 platforms that use the APIC (namely Fiasco.OC, NOVA, and hw_x86_64)
this simple ACPI parser traverses the ACPI tables and reports device-resource
information (e.g., interrupt lines of PCI devices).
:'libports/src/app/acpica':
:_os/src/app/smbios_decoder/_:
A component that parses SMBIOS information on x86 platforms and makes the
result available as a report.
:_libports/src/app/acpica/_:
In addition to our ACPI base driver, the acpica component uses the
ACPICA library to provide access to dynamic functions like battery
states, events (e.g., notebook lid close and power buttons), as well
@@ -93,91 +102,71 @@ UART devices
The UART device drivers implement the UART-session interface.
:'os/src/drivers/uart/spec/pl011':
:_os/src/drivers/uart/spec/pbxa9/_:
Driver for the PL011 UART as found on many ARM-based platforms.
:'os/src/drivers/uart/spec/i8250':
:_os/src/drivers/uart/spec/x86/_:
Driver for the i8250 UART as found on PC hardware.
:'os/src/drivers/uart/spec/omap4':
Driver for the UART as found on OMAP4-based hardware.
:'os/src/drivers/uart/spec/exynos5':
Driver for the UART as found on Exynos-5-based hardware.
Framebuffer and input drivers
=============================
Framebuffer and input drivers implement the framebuffer-session interface and
input-session interfaces respectively.
Framebuffer and input drivers are implemented as clients of the
capture-session and event-session interfaces respectively.
:'os/src/drivers/input/dummy':
Pseudo input driver without accessing any hardware. This component is useful
to resolve a dependency from an input session for scenarios where no user
input is required.
:'os/src/drivers/input/spec/ps2/x86':
:_os/src/drivers/ps2/x86/_:
Driver for the 'i8042' PS/2 controller as found in x86 PCs. It supports both
mouse (including ImPS/2, ExPS/2) and keyboard.
:'os/src/drivers/input/spec/ps2/pl050':
:_os/src/drivers/ps2/pl050/_:
Driver for the PL050 PS/2 controller as found on ARM platforms such as
VersatilePB. The physical base address used by the driver is obtained at
compile time from a header file called 'pl050_defs.h'. The version of the
VersatilePB platform can be found at 'os/include/platform/vpb926/' and
compile time from a header file called _pl050_defs.h_. The version of the
VersatilePB platform can be found at _os/include/platform/vpb926/_ and
is made available to the driver via the SPECS machinery of the Genode build
system.
:'os/src/drivers/input/spec/imx53':
Input driver for Egalaxy touchscreen and Freescale's MPR121
capacitative touch buttons on i.MX53.
:'libports/src/drivers/framebuffer/vesa':
:_libports/src/drivers/framebuffer/vesa/_:
Driver using VESA mode setting on x86 PCs. For more information, please refer
to the README file in the driver directory.
:'libports/src/drivers/framebuffer/boot':
:_libports/src/drivers/framebuffer/boot/_:
Driver for boot-time initialized framebuffers (e.g., UEFI GOP)
discovered from the 'platform_info' ROM
:'os/src/drivers/framebuffer/spec/pl11x':
:_os/src/drivers/framebuffer/pl11x/_:
Driver for the PL110/PL111 LCD display.
:'os/src/drivers/framebuffer/spec/omap4':
Driver for HDMI output on OMAP4 SoCs.
:'os/src/drivers/framebuffer/spec/exynos5':
Driver for HDMI output on Exynos-5 SoCs.
:'os/src/drivers/framebuffer/spec/imx53':
:_os/src/drivers/framebuffer/imx53/_:
Driver for LCD output on i.MX53 SoCs.
:'os/src/drivers/framebuffer/spec/rpi':
:_os/src/drivers/framebuffer/rpi/_:
Driver for the HDMI output of the Raspberry Pi.
:'os/src/drivers/framebuffer/spec/sdl':
:_os/src/drivers/framebuffer/sdl/_:
Serves as both framebuffer and input driver on Linux using libSDL. This
driver is only usable on the Linux base platform.
:'os/src/drivers/gpu/intel':
Intel Graphics GPU multiplexer for Broadwell and newer.
:_os/src/drivers/gpu/intel/_:
An experimental Intel Graphics GPU multiplexer for Broadwell and newer.
:'dde_linux/src/drivers/framebuffer/intel':
:_dde_linux/src/drivers/framebuffer/intel/_:
Framebuffer driver for Intel i915 compatible graphic cards based on
the Linux Intel KMS driver.
:'dde_linux/src/drivers/usb':
USB driver that makes USB HID and USB storage devices available as input
sessions and block session respectively. For examples of using this driver,
refer to the run scripts at 'dde_linux/run/usb_hid' and
'dde_linux/run/usb_storage'.
:_dde_linux/src/drivers/usb_host/_:
USB host-controller driver that provides an USB session interface to
USB drivers.
:_dde_linux/src/drivers/usb_hid/_:
USB Human Interface Device driver using the USB session interface.
Timer drivers
=============
The timer driver located at 'os/src/drivers/timer' implements the timer-session
The timer driver located at _base/src/timer/_ implements the timer-session
interface. Technically, it is both a device driver (accessing a timer
device) and a resource multiplexer (supporting multiple timer-session clients
at the same time). Depending on the base platform, the implementation uses
@@ -198,13 +187,13 @@ Audio drivers
=============
Audio drivers implement the Audio_out session interface defined at
'os/include/audio_out_session/' for playback and optionally the audio_in
_os/include/audio_out_session/_ for playback and optionally the audio_in
interface for recording.
:'os/src/drivers/audio/spec/linux':
:_os/src/drivers/audio/spec/linux/_:
Uses ALSA as back-end on the Linux base platform and supports only playback.
:'dde_bsd/src/drivers/audio':
:_dde_bsd/src/drivers/audio/_:
Sound drivers ported from OpenBSD. Currently, the repository
includes support for Intel HD Audio as well as for Ensoniq AudioPCI
(ES1370) compatible sound cards.
@@ -214,121 +203,111 @@ Block drivers
=============
All block drivers implement the block-session interface defined at
'os/include/block_session/'.
_os/include/block_session/_.
:'os/src/drivers/sd_card/spec/pl180':
:_os/src/drivers/sd_card/spec/pl180/_:
Driver for SD-cards connected via the PL180 device as found on the PBX-A9
platform.
:'os/src/drivers/sd_card/spec/omap4':
Driver for SD-cards connected to the SD-card controller of the OMAP4 SoC.
:'os/src/drivers/sd_card/spec/exynos5':
Driver for SD-cards and eMMC connected to Exynos-5-based platforms.
:'os/src/drivers/sd_card/spec/imx53':
:_os/src/drivers/sd_card/spec/imx53/_:
Driver for SD-cards connected to the Freescale i.MX53 platform like the
Quick Start Board or the USB armory device.
:'os/src/drivers/sd_card/spec/rpi':
:_os/src/drivers/sd_card/spec/rpi/_:
Driver for SD-cards connected to the Raspberry Pi.
:'dde_linux/src/drivers/usb':
USB driver that makes USB storage devices available as block sessions.
For an example of using this driver, refer to the run script at
'dde_linux/run/usb_storage'.
:'os/src/drivers/ahci':
:_os/src/drivers/ahci/_:
Driver for SATA disks and CD-ROMs on x86 PCs.
:'os/src/drivers/usb_block':
USB Mass Storage Bulk-Only driver using the USB session interface.
:_os/src/drivers/nvme/_:
Driver for NVMe block devices on x86 PCs.
:_os/src/drivers/usb_block/_:
USB Mass Storage Bulk-Only driver using the USB session interface and provides
a block-session interface.
Network interface drivers
=========================
All network interface drivers implement the NIC session interface
defined at 'os/include/nic_session'.
defined at _os/include/nic_session/_.
:'os/src/drivers/nic/spec/linux':
:_os/src/drivers/nic/spec/linux/_:
Driver that uses a Linux tap device as back end. It is only useful on the
Linux base platform.
:'os/src/drivers/nic/spec/lan9118':
:_os/src/drivers/nic/spec/lan9118/_:
Native device driver for the LAN9118 network adaptor as featured on the
PBX-A9 platform.
:'os/src/drivers/nic/gem':
Device driver for Cadence EMAC PS network adaptor as featured on the
Xilinx Zynq.
:'dde_ipxe/src/drivers/nic':
:_dde_ipxe/src/drivers/nic/_:
Device drivers ported from the iPXE project. Supported devices are Intel
E1000 and pcnet32.
:'dde_linux/src/drivers/wifi':
:_dde_linux/src/drivers/wifi/_:
The wifi_drv component is a port of the Linux mac802.11 stack, including the
iwlwifi driver. It enables the use of Intel Wireless 6xxx and 7xxx cards.
:'dde_linux/src/drivers/usb':
For the OMAP4 platform, the USB driver contains the networking driver.
:_dde_linux/src/drivers/usb_net/_:
USB network driver using the USB session interface.
:'dde_linux/src/drivers/nic/fec':
:_dde_linux/src/drivers/nic/fec/_:
Driver for ethernet NICs of the i.MX SoC family.
General-purpose I/O drivers
===========================
:'os/src/drivers/gpio/spec/omap4':
Driver for accessing the GPIO pins of OMAP4 platforms.
:'os/src/drivers/gpio/spec/imx53':
:_os/src/drivers/gpio/spec/imx53/_:
Driver for accessing the GPIO pins of i.MX53 platforms.
:'os/src/drivers/gpio/spec/rpi':
:_os/src/drivers/gpio/spec/rpi/_:
Driver for accessing the GPIO pins of Raspberry Pi platforms.
:'os/src/drivers/gpio/spec/exynos5':
Driver for accessing the GPIO pins of Exynos4 platforms, e.g.,
Odroid-X2.
Resource multiplexers
#####################
By convention, resource multiplexers are located at the 'src/server'
By convention, resource multiplexers are located at the _src/server/_
subdirectory of a source repository.
:Framebuffer and input: The framebuffer and input session interfaces can be
multiplexed using the Nitpicker GUI server, which allows multiple clients to
create and manage rectangular areas on screen. Nitpicker uses one input
session and one framebuffer session as back end and, in turn, provides
so-called nitpicker sessions to one or multiple clients. Each nitpicker
session contains a virtual framebuffer and a virtual input session. Nitpicker
(including a README file) is located at 'os/src/server/nitpicker'.
:Framebuffer and input: Framebuffer and input devices can be multiplexed using
the Nitpicker GUI server, which allows multiple clients to create and manage
rectangular areas on screen. Nitpicker serves as broker between input
devices, output devices, and graphical applications. It provides an event
service for input drivers, a capture service for output drivers, and a GUI
service for the applications. Each GUI session contains a virtual
framebuffer and a virtual input interface. Nitpicker (including a README
file) is located at _os/src/server/nitpicker/_.
:Audio output: The audio mixer located at 'os/src/server/mixer' enables
:Audio output: The audio mixer located at _os/src/server/mixer/_ enables
multiple clients to use the audio-out interface. The mixing is done by simply
adding and clamping the signals of all present clients.
:Networking: The NIC bridge located at 'os/src/server/nic_bridge' multiplexes
:Networking: The NIC bridge located at _os/src/server/nic_bridge/_ multiplexes
one NIC session to multiple virtual NIC sessions using a proxy-ARP
implementation. Each client has to obtain a dedicated IP address visible to
the physical network. DHCP requests originating from the virtual NIC sessions
are delegated to the physical network.
:Block: The block-device partition server at 'os/src/server/part_block' reads
The NIC router located at _os/src/server/nic_router/_ multiplexes one NIC
session to multiple virtual NIC sessions by applying network address
translation (NAT).
:Block: The block-device partition server at _os/src/server/part_block/_ reads
the partition table of a block session and exports each partition found as
separate block session. For using this server, please refer to the run
script at 'os/run/part_block'.
script at _os/run/part_block.run_.
:File system: The FAT file-system service allows multiple clients to
concurrently access the same FAT-formatted block device. It is located
at 'libports/src/server/fatfs_fs' and supports FAT, FAT32, and exFAT.
:File system: The VFS file-system server allows multiple clients to
concurrently access the same virtual file system. It is located at
_os/src/server/vfs/_. The VFS can be assembled out of several builtin
file-system types (like a RAM file system, or pseudo file systems for
various Genode session interfaces) as well as external plugins such as rump
(mounting file systems supported by the NetBSD kernel).
:Terminal: The terminal_mux service located at gems/src/server/terminal_mux
:Terminal: The terminal_mux service located at _gems/src/server/terminal_mux/_
is able to provide multiple terminal sessions over one terminal-client
session. The user can switch between the different sessions using a keyboard
shortcut, which brings up an ncurses-based menu.
@@ -340,214 +319,214 @@ Protocol stacks
Protocol stacks come either in the form of separate components that translate
one session interface to another, or in the form of libraries.
Separate components:
Separate components
===================
:'os/src/server/nit_fb':
Translates a nitpicker session to a pair of framebuffer and input sessions.
Each 'nit_fb' instance is visible as a rectangular area on screen presenting
:_os/src/server/gui_fb/_:
Translates a GUI session to a pair of framebuffer and input sessions.
Each 'gui_fb' instance is visible as a rectangular area on screen presenting
a virtual frame buffer. The area is statically positioned. For more
information, please refer to 'os/src/server/nit_fb/README'.
information, please refer to _os/src/server/gui_fb/README_.
:'gems/src/server/wm':
Window manager that implements the nitpicker session interface but manages
:_gems/src/server/wm/_:
Window manager that implements the GUI session interface but manages
each client view as a separate window. The window decorations are provided
by a so-called decorator (e.g., 'gems/src/app/decorator'). The behaviour
by a so-called decorator (e.g., _gems/src/app/decorator/_). The behaviour
is defined by a so-called window layouter such as the floating window
layouter located at 'gems/src/app/floating_window_layouter/'.
layouter located at _gems/src/app/floating_window_layouter/_.
:'demo/src/server/liquid_framebuffer':
Implements the same translation as 'nit_fb' but by presenting an interactive
:_demo/src/server/liquid_framebuffer/_:
Implements the same translation as 'gui_fb' but by presenting an interactive
window rather than a statically positioned screen area.
:'os/src/server/tar_rom':
:_os/src/server/tar_rom/_:
Provides each file contained in a tar file obtained via Genode's ROM session
as separate ROM session.
:'os/src/server/iso9660':
:_os/src/server/iso9660/_:
Provides each file of an ISO9660 file system accessed via a block session as
separate ROM session.
:'os/src/server/ram_fs':
A file-system implementation that keeps all data in memory.
:'dde_rump/src/server/rump_fs':
A file-system server that contains various file-systems ported from the
NetBSD kernel.
:'os/src/server/lx_fs':
:_os/src/server/lx_fs/_:
A file system server that makes the file system of a Linux base platform
available to Genode.
:'os/src/server/trace_fs':
A pseudo file system that can be used as a front end to core's TRACE
service.
:_os/src/server/vfs_block/_:
Provides the content of a file obtained from a VFS as a block session,
similar to the loop-mount mechanism on Linux
:'os/src/server/rom_block':
Provides the content of a ROM file as a block session, similar to the
loop-mount mechanism on Linux
:'os/src/server/ram_block':
Provides the content of a RAM dataspace as a block session. In contrast
to 'rom_block', this server provides a writeable block device.
:'os/src/server/terminal_log':
:_os/src/server/terminal_log/_:
Adapter for forwarding LOG messages to a terminal session.
:'os/src/server/log_terminal':
:_os/src/server/log_terminal/_:
Adapter for forwarding terminal output to a LOG session.
:'os/src/server/fs_log':
:_os/src/server/fs_log/_:
Adapter that writes LOG messages to files on a file system.
:'demo/src/server/nitlog':
Provides a LOG session, printing log output on screen via a nitpicker
session.
:_demo/src/server/nitlog/_:
Provides a LOG session, printing log output on screen via a GUI session.
:'os/src/app/rom_logger':
:_os/src/app/rom_logger/_:
The rom_logger component requests a ROM session and writes the
content of the ROM dataspace to the LOG.
:'os/src/server/rom_filter':
:_os/src/server/rom_filter/_:
The ROM filter provides a ROM module that depends on the content of
other ROM modules steered by the filter configuration, e.g., dynamic
switching between configuration variants dependent on the state of
the system.
:'os/src/server/vfs':
A file-system server using the VFS library and plugins as backend.
:'os/src/server/log_terminal':
:_os/src/server/log_terminal/_:
Forwards terminal output to a LOG session.
:'gems/src/server/file_terminal':
:_gems/src/server/file_terminal/_:
Provides terminal sessions that target files on a file system.
:'gems/src/server/terminal':
:_gems/src/server/terminal/_:
Provides a terminal session via a graphical terminal using a framebuffer
session and an input session.
:'gems/src/server/tcp_terminal':
:_gems/src/server/tcp_terminal/_:
Provides one or multiple terminal sessions over TCP connections.
For further information, refer to 'gems/src/server/tcp_terminal/README'.
For further information, refer to _gems/src/server/tcp_terminal/README_.
:'os/src/server/terminal_crosslink':
:_os/src/server/terminal_crosslink/_:
The terminal crosslink service allows to terminal clients to talk to each
other.
:'gems/src/server/http_block':
:_gems/src/server/http_block/_:
A block service that fetches a virtual block device over the network from
a HTTP server.
:'os/src/server/fs_rom':
:_os/src/server/fs_rom/_:
A ROM service that translates the 'File_system' session interface to the
'ROM' session' interface. Each request for a ROM file is handled by looking
up an equally named file on the file system.
Please refer to 'os/src/server/fs_rom' for more information.
Please refer to _os/src/server/fs_rom/_ for more information.
:'os/src/server/dynamic_rom':
For use cases where ROMs are known to be static, the
_os/src/server/cached_fs_rom/_ can be considered as a faster alternative to
the regular 'fs_rom' server. Note that 'cached_fs_rom' is not supported
in base-linux though.
:_os/src/server/chroot/_:
An intermediate file-system server that makes a sub directory of a file
system available as the root of a file system handed out to its client.
:_os/src/server/dynamic_rom/_:
A simple ROM service that provides ROM modules that change in time according
to a configured timeline.
:'os/src/server/report_rom':
:_os/src/server/report_rom/_:
A service that implements both the report session interface and the ROM
session interface. It reflects incoming reports as ROM modules.
:'os/src/server/fs_report':
:_os/src/server/fs_report/_:
Report server that writes reports to file-systems
:'os/src/server/clipboard':
:_os/src/server/clipboard/_:
This component is both a report service and a ROM service. The
clients of the report service can issue new clipboard content, which
is then propagated to the clients of the ROM service according to a
configurable information-flow policy.
:'ports/src/app/openvpn':
OpenVPN enables access to remote network resources through a secure tunnel
by providing an encrypted connection to a remote host. It is plugged between
NIC server (such as a network driver) and NIC client.
:_os/src/server/event_filter/_:
A component that transforms and merges input events from multiple sources
into a single event stream.
:'os/src/server/input_merger':
A component that merges input events from multiple sources into a single
stream.
:'libports/src/server/acpi_input':
:_libports/src/app/acpi_event/_:
A component that transforms ACPI events into Genode input events.
:'gems/src/server/nit_fader':
A wrapper for nitpicker's session interface that applies alpha-blending to
the of views a nitpicker client.
:_gems/src/server/gui_fader/_:
A wrapper for nitpicker's GUI session interface that applies alpha-blending
to the of views a GUI client.
Libraries:
:_os/src/server/black_hole/_:
Mockup implementation of Genode session interfaces.
:'libports/lib/mk/libc':
VFS plugins
===========
VFS plugins are file-system drivers in the form of shared libraries that
implement the VFS-plugin interface. They can be combined with any application
based on Genode's C runtime, with the VFS server, and with non-POSIX
components that use the Genode's VFS library directly.
:_gems/src/lib/vfs/trace/_:
A VFS plugin that makes core's TRACE service accessible as a pseudo
file system.
:_gems/src/lib/vfs/import/_:
A VFS plugin that pre-populates a VFS with initial content.
:_gems/src/lib/vfs/pipe/_:
A VFS plugin that provides bi-directional pipes for exchanging streamed
data between components.
:_gems/src/lib/vfs/ttf/_:
A VFS plugin that makes rendered pixel data of the glyphs of Truetype fonts
available as a pseudo file system.
:_libports/src/lib/vfs/jitterentropy/_:
A VFS plugin that provides random numbers based on the jitter of executing
CPU instructions.
:_libports/src/lib/vfs/lwip/_:
A VFS plugin that uses the light-weight IP (lwIP) stack to provide a
network socket interface as a pseudo file system.
:_dde_linux/src/lib/vfs/lxip/_:
A VFS plugin that uses the TCP/IP stack ported from the Linux kernel to
provide a network socket interface as a pseudo file system.
:_libports/src/lib/vfs/fatfs/_:
A VFS plugin that allows for the mounting of FAT-formatted block devices.
:_dde_rump/src/lib/vfs/rump/_:
A VFS plugin that enables the use of NetBSD's file-system drivers such
as ext2 or msdos.
Libraries
=========
:_libports/lib/mk/libc/_:
C runtime ported from FreeBSD.
:'libports/lib/mk/libc_lwip_nic_dhcp':
Translates the BSD socket API to a NIC session using the lwIP stack.
:'dde_linux/lib/mk/libc_lxip':
Translates the BSD socket API to a NIC session using the Linux TCP/IP stack.
:'libports/lib/mk/libc_ffat':
Accesses files on a block device that contains a FAT32 file system.
:'libports/lib/mk/libc_fuse_exfat':
Accesses files on a block device that contains an exFAT file system.
:'libports/lib/mk/libc_fuse_ext2':
Accesses files on a block device that contains an ext2 file system.
:'libports/lib/mk/libc_terminal':
Connects the standard input and output from/to Genode's terminal session
interface.
:'libports/lib/mk/stdcxx':
:_libports/lib/mk/stdcxx/_:
Standard C++ library
:'libports/lib/mk/mesa_api':
:_libports/lib/mk/mesa_api/_:
Mesa OpenGL API with backends for software rasterization (egl_swrast)
and Intel Graphics (egl_i965)
:'libports/lib/mk/pthread':
Subset of the POSIX thread and semaphore API.
:'libports/lib/mk/python':
Runtime of the Python scripting language.
:'libports/lib/mk/mupdf':
:_libports/lib/mk/mupdf/_:
PDF rendering engine.
:'libports/lib/mk/sdl':
Translates the libSDL API to framebuffer and input sessions.
:'libports/lib/mk/ncurses':
:_libports/lib/mk/ncurses/_:
Library for implementing pseudo-graphical applications (i.e., VIM) that
run on a text terminal.
:'libports/lib/mk/avcodec':
A library for video decoding, conversion, and streaming.
:_libports/lib/mk/qt5_*/_:
Qt5 framework, using GUI session and NIC session as back end.
:'libports/lib/mk/lua':
Runtime for the Lua scripting language.
:'libports/lib/mk/qt5_*':
Qt5 framework, using nitpicker session and NIC session as back end.
:'libports/lib/mk/vfs_jitterentropy.mk':
:_libports/lib/mk/vfs_jitterentropy.mk_:
A VFS plugin that makes a jitter-based random-number generator available
as a file within the process-local VFS.
:'libports/lib/mk/libarchive.mk':
:_libports/lib/mk/libarchive.mk_:
Library providing a common interface to a variety of archive
formats.
:'libports/lib/mk/lz4.mk':
:_libports/lib/mk/lz4.mk_:
Library for processing LZ4 lossless compression archives.
:'libports/lib/mk/liblzma.mk':
:_libports/lib/mk/liblzma.mk_:
Library for processing LZMA archives.
:'libports/lib/mk/libgcrypt.mk':
:_libports/lib/mk/libgcrypt.mk_:
GnuPG library for OpenPGP processing, e.g., signature verification.
@@ -555,103 +534,96 @@ Applications
############
Applications are Genode components that use other component's services but
usually do not provide services. They are typically located in the 'src/app/'
usually do not provide services. They are typically located in the _src/app/_
subdirectory of a repository. Most applications come with README files
located in their respective directory.
:'gems/src/app/backdrop':
Nitpicker client application that sets a composition of PNG images as
desktop background.
:_gems/src/app/backdrop/_:
GUI client application that sets a composition of PNG images as desktop
background.
:'demo/src/app/launchpad':
:_demo/src/app/launchpad/_:
Graphical application for interactively starting and killing subsystems.
:'gems/app/launcher': Graphical launcher of Genode subsystems.
:'os/app/cli_monitor': Command-line-based launcher of Genode subsystems.
:'demo/src/app/scout':
:_demo/src/app/scout/_:
Graphical hypertext browser used for Genode's default demonstration scenario.
:'libports/src/test/mesa_demo':
Example programs for using the Mesa OpenGL graphics stack.
:'ports/src/app/arora':
Arora is a Qt-based web browser using the Webkit engine.
:'ports/src/app/gdb_monitor':
:_ports/src/app/gdb_monitor/_:
Application that allows the debugging of a process via GDB over a remote
connection.
:'libports/src/app/qt5/qt_launchpad':
:_libports/src/app/qt5/qt_launchpad/_:
Graphical application starter implemented using Qt.
:'libports/src/app/qt5/examples/':
:_libports/src/app/qt5/examples/_:
Several example applications that come with Qt.
:'os/src/app/sequence':
:_os/src/app/sequence/_:
Simple utility to serialize the execution of multiple components
:'ports/src/noux-pkg':
:_ports/src/noux-pkg/_:
Ports of popular commandline-based Unix software such as VIM, bash,
coreutils, binutils, gcc, findutils, and netcat. The programs are supposed
to be executed within the Noux runtime environment.
:'ports/src/app/lighttpd':
:_ports/src/app/lighttpd/_:
Lighttpd is a fast and feature-rich web server. The port of lighttpd uses
a file-system session to access the website content and the web-server
configuration.
:'os/src/app/trace_logger':
:_os/src/app/trace_logger/_:
Convenient, runtime-configurable frontend to the tracing facility.
:'os/src/app/rom_reporter':
:_os/src/app/rom_reporter/_:
The ROM-reporter component requests a ROM session and reports the
content of the ROM dataspace to a report session with the same label
as the ROM session.
:'os/src/app/log_core':
:_os/src/app/log_core/_:
Component transforming core and kernel output to Genode LOG output.
Package-management components
=============================
:'gems/src/app/depot_query':
:_gems/src/app/depot_query/_:
Tool for querying subsystem information from a depot.
:'gems/src/app/depot_download_manager':
:_gems/src/app/depot_download_manager/_:
Tool for managing the download of depot content.
:'gems/src/app/depot_deploy':
:_gems/src/app/depot_deploy/_:
Subsystem init configuration generator based on blueprints.
:'libports/src/app/fetchurl':
:_libports/src/app/fetchurl/_:
A runtime-configurable frontend to the libcURL library for
downloading content.
:'libports/src/app/extract':
:_libports/src/app/extract/_:
Tool for extracting archives using libarchive.
:'ports/src/app/verify':
:_ports/src/app/verify/_:
This component verifies detached OpenPGP signatures using libgcrypt.
Runtime environments
####################
:'ports/src/noux': Noux is an experimental implementation of a UNIX-like API
that enables the use of unmodified command-line based GNU software. For using
noux, refer to the run script 'ports/run/noux.run'.
:'ports/src/app/seoul': Seoul is a virtual-machine monitor developed for
:_ports/src/app/seoul/_: Seoul is a virtual-machine monitor developed for
the use with the NOVA platform. It virtualizes 32bit x86 PC hardware
including various peripherals.
:'os/src/server/loader': A service that allows the creation and destruction
:_os/src/server/loader/_: A service that allows the creation and destruction
of Genode subsystems via a session interface. For further information,
refer to 'os/src/server/loader/README'.
refer to _os/src/server/loader/README_.
:'ports/src/app/dosbox': A port of DosBox for executing DOS software.
:_ports/src/virtualbox5/_: VirtualBox running on top of the NOVA hypervisor.
:'ports/src/virtualbox': VirtualBox running on top of the NOVA hypervisor.
:_os/src/server/vmm/_: A virtual machine monitor that is based on
hardware-assisted virtualization of ARM platforms. It is supported on
the base-hw kernel only.
:_os/src/server/cpu_balancer/_: The CPU balancer intercepts the interaction
of components with core's low-level services to migrate threads dynamically
between CPU cores.

2
doc/contributions.txt
View File

@@ -140,7 +140,7 @@ permission to let Genode Labs redistribute his contributions under non-AGPLv3
licenses. This permission is granted by signing the Genode Contributors
Agreement:
:[http:gca.pdf - Genode Contributor's Agreement]:
:[https://genode.org/community/gca.pdf - Genode Contributor's Agreement]:
Genode Contributor's Agreement (GCA)
By signing the GCA, you don't lose any rights for your contribution. However,

21
doc/getting_started.txt
View File

@@ -50,17 +50,26 @@ This creates a new build directory for building x86_64 binaries in './build'.
The build system creates unified binaries that work on the given
architecture independent from the underlying base platform, in this case Linux.
To give Genode a try, build and execute a simple demo scenario via:
Now change into the fresh build directory:
! cd build/x86_64
! make KERNEL=linux run/demo
Please uncomment the following line in 'etc/build.conf' to make the
build process as smooth as possible.
! RUN_OPT += --depot-auto-update
To give Genode a try, build and execute a simple demo scenario via:
! make KERNEL=linux BOARD=linux run/demo
By invoking 'make' with the 'run/demo' argument, all components needed by the
demo scenario are built and the demo is executed. This includes all components
which are implicitly needed by the base platform. The base platform that the
components will be executed upon on is selected via the 'KERNEL' variable. If
you are interested in looking behind the scenes of the demo scenario, please
refer to 'doc/build_system.txt' and the run script at 'os/run/demo.run'.
components will be executed upon on is selected via the 'KERNEL' and 'BOARD'
variables. If you are interested in looking behind the scenes of the demo
scenario, please refer to 'doc/build_system.txt' and the run script at
'os/run/demo.run'.
Using platforms other than Linux
@@ -112,7 +121,7 @@ steps are required:
# Uncomment the following line in 'x86_32/etc/build.conf'
! REPOSITORIES += $(GENODE_DIR)/repos/libports
# Build and execute the demo using Qemu
! make -C build/x86_32 KERNEL=okl4 run/demo
! make -C build/x86_32 KERNEL=okl4 BOARD=pc run/demo
The procedure works analogously for the other base platforms. You can, however,
reuse the already created build directory and skip its creation step if the

649
doc/news.txt
View File

@@ -4,6 +4,625 @@
===========
Genode OS Framework release 21.08 | 2021-08-31
##############################################
| The highlights of Genode 21.08 are revamped GPU support as well as new
| drivers for the Pinephone and MNT-Reform laptop based on a new streamlined
| approach for porting Linux kernel code. Further topics range from VirtualBox
| improvements, over media playback in the native web browser, to LTE
| connectivity in Sculpt OS.
For complex driver stacks, Genode largely relies on code ported from other
operating systems. The Linux kernel plays a special role because - being the
basis for Android - it is the de-facto reference for driving the peripherals
of most ARM SoCs. Up to now, however, the porting efforts of driver code from
Linux to Genode used to be a time-intensive affair, which forced a narrow
focus on very few SoCs on us. With the streamlined porting approach introduced
with the new release, we become able to dramatically reduce the costs,
creating the prospect of a much broader hardware support. The first success
stories of the new way of porting are added graphics drivers for the Pinephone
and the MNT-Reform laptop, a network driver for the Pine-A64-LTS board, and an
SD-card driver for the MNT-Reform.
The second spotlight of the release is the largely revamped support for Intel
GPUs. In contrast to our experimental GPU-related work of the past, we have
now identified a way to cleanly integrate GPU support into the GUI
architecture of sophisticated Genode systems such as Sculpt OS. This work is
accompanied with an up-to-date version of the Mesa library stack. In
combination with the improvements of our custom GPU multiplexer, we are now on
a good track to make the use of hardware-accelerated graphics a commodity on
Genode.
Even though most topics of the current release revolve around low-level
driver-related work, the new version improves higher-level functionality as
well. In particular, it adds the modular integration of mobile-data
connectivity to Sculpt OS and enables media playback for our port of the
Chromium web engine. Those and more topics are described in the detailed
[https:/documentation/release-notes/21.08 - release documentation of version 21.08...]
Genode OS Framework release 21.05 | 2021-05-31
##############################################
| Version 21.05 introduces webcam support, features an encrypted file vault
| for Sculpt OS, and adds new drivers for the i.MX8 SoC and Pine-A64.
| Furthermore, it is accompanied with a new tool chain based on GCC 10
| and new guidance documentation for driver development.
Device-driver support remains a prominent topic for almost every release.
The just released version 21.05 is no exception, touching hardware topics
ranging from USB, over I2C on i.MX8, over GPIO on Pine-A64, to networking on
RISC-V. Until now, such nitty-gritty driver-development work has been pursued
almost exclusively by the core team at Genode Labs. To encourage developers
outside the inner circle to join the fun, the release is accompanied with the
initial version of a comprehensive guidance documentation for
hardware-related topics.
Feature-wise the two highlights of the new version are webcam support and an
easy-to-use encrypted file store based on our custom CBE block encrypter.
As detailed by the release documentation, both features leverage Genode's
architecture in unique ways to attain high flexibility without an inflation of
complexity.
The release is rounded up by a new tool chain based on GCC 10 and Binutils
2.36, profound performance optimizations, framework refinements, and new
assistive tooling for porting Linux drivers. All the details of the new
version are covered by the official
[https:/documentation/release-notes/21.05 - release documentation of version 21.05...]
Sculpt OS 21.03 boots now in 2.5 seconds | 2021-05-03
#####################################################
| We have released an updated Sculpt 21.03 image featuring several
| hardware-compatibility tweaks and performance improvements.
Since the official Sculpt OS 21.03 release end of March, we continued our
efforts with refining the user experience and broadening hardware
compatibility, thanks to the user feedback we received. We have now released
an updated system image that includes those refinements as well as several
performance optimizations that boost the boot time of Sculpt to less than 2.5
seconds from the first life sign of the kernel to the graphical user
interface (measured on a 5-years old Lenovo x250 laptop).
You can get the new version of the system image named sculpt-21-03b at the
[https://genode.org/download/sculpt - Sculpt OS download] page.
It is binary compatible with the original release version. So you can use it
as a drop-in-replacement.
For manually reproducing the image, please refer to the corresponding
[https://github.com/chelmuth/genode/commits/sculpt_21_03b - Git branch].
Sculpt OS release 21.03 | 2021-03-24
####################################
| Version 21.03 of the Sculpt operating system makes the system resilient
| against classes of driver failures, adds configurable real-time priorities,
| and introduces interfaces for screen capturing and user-event injection.
Sculpt OS 21.03 incorporates the many improvements of the latest two Genode
releases. Thanks to Genode's concept of
[https://genode.org/documentation/release-notes/21.02#Pluggable_network_device_drivers - pluggable device drivers],
the system has reached a new level of robustness against malfunctioning
drivers. For example, if the Intel graphics driver trips over an unsupported
external display, the driver gets automatically restarted while all graphical
applications keep running. Or as another example, should the overly complex
Wifi driver have a hick-up, it can be restarted with a simple mouse click
without harming the networking stacks running on top.
Even though Genode supports static-priority scheduling since more than a
decade, Sculpt did not make this feature available to end users so far. The
new version changes that. For each component, the user can now take a
deliberate decision about the hard scheduling priority, e.g., prioritizing
latency-critical multi-media applications over computational workloads or
virtual machines.
Speaking of workloads, to push the limits of what is possible with Sculpt OS,
the new version introduces additional interfaces that can be assigned to
components. First, it has become possible to redirect the interaction of a
component with the kernel through another component, thereby enabling features
like dynamic CPU-load balancing to be implemented as plain user-level
services. Second, there are new interfaces for capturing the screen and for
injecting input events. The latter interfaces pave the ground for virtual
keyboards, screen-sharing application, or remote administration scenarios.
Under the hood, there are plenty of improvements that make the life of
Sculpt users better. The keyboard layout can now be picked from a menu.
The Chromium-based Falkon web browser runs circles around the previous
version. Menu items and file lists appear nicely sorted. Terminal windows
immediately respond to global font-size changes. On modern Intel machines,
Sculpt leverages Intel Hardware P-states (HWP) for power and thermal
management now. You can find an illustrated tour of these and more changes in
a dedicated
[https://genodians.org/nfeske/2021-03-24-sculpt-os - article at Genodians.org].
The updated [https://genode.org/documentation/articles/sculpt-21-03 - manual]
goes into detail about the use of the new system.
The ready-to-use system image for version 21.03 is available at the
[https://genode.org/download/sculpt - Sculpt download page].
Genode OS Framework release 21.02 | 2021-02-25
##############################################
| The highlights of version 21.02 are the addition of VirtualBox 6,
| mobile-data connectivity via LTE, pluggable network drivers, initial
| support for the Pine-A64-LTS board, and revived work on RISC-V.
Many topics of the current release draw a connection to our overarching goal
to use Genode on the Pinephone by the end of the year. Besides the obvious
steps of enabling the hardware - starting with the Pine-A64-LTS board - the
release introduces mobile-data connectivity as a Genode feature, and changes
the network-driver architecture in anticipation of dynamic power-management
schemes.
For PC hardware, the flagship feature of version 21.02 is the addition of
VirtualBox 6, giving us the prospect to eventually replace the aging port of
VirtualBox 5. Speaking of VirtualBox, the release comes with profound
improvements of the USB-device pass-through abilities, most importantly
covering audio headsets.
Besides these prominent features, the new version comes with many further
improvements. Just to name a few, virtual machines on ARM have become
able to provide VirtIO-block devices to guests, named pipes can now
be used to connect components, Genode's RISC-V support received an
update to ISA spec 1.10, and OpenSSL has been bumped to version 1.1.1.
For the full story, please refer to the
[https:/documentation/release-notes/21.02 - release documentation of version 21.02...]
Road Map for 2021 | 2021-01-15
##############################
| In 2021, we plan to bring Genode to the Pinephone, advance the framework's
| GPU support, and focus on development workflows.
During the annual road-map discussion on Genode's public
[https://genode.org/community/mailing-lists - mailing list],
the following hot topics for this year emerged.
First and most inspiring for many Genode developers, we aspire to have
Genode running on the Pinephone with basic feature-phone functionality by the
end of the year. Since this will involve substantial device-driver-related
developments, the team will take this line of work as an opportunity to
advance the tooling and workflows for carrying out such tasks. This, in turn,
will hopefully ease the on-boarding of new driver developers in the future.
Closely related to the Pinephone scenario, the project will make optimizations
a top priority this year. The opportunities are plenty, ranging from
micro-optimizations, over API refinements, to architectural changes if
needed.
Another recurring topic is the request for GPU support, which is required
by many modern workloads such as video conferencing or streaming on mobile
device. Therefore, we will revamp our past developments of GPU multiplexing
on Intel hardware while also starting the investigation of GPUs on ARM-based
devices.
More information about our review of the past year, this year's focus, and a
rough schedule are presented at our official
[https:/about/road-map - road-map page].
Genode OS Framework release 20.11 | 2020-11-27
##############################################
| Genode 20.11 brings Sculpt OS to 64-bit ARM hardware, introduces dynamic
| CPU-load balancing, and enables multicore virtualization on ARM. Driver-wise,
| the release improves audio on PC hardware, and adds VirtIO networking support.
ARM 64-bit has been a recurring theme of the Genode releases this year and the
just released version 20.11 is no exception. We are proud to announce that our
Genode-based custom general-purpose OS called Sculpt has come to life on
64-bit ARM hardware, namely the NXP i.MX8 EVK board. This is the result of
intensive work on the framework's driver architecture for ARM and several
SoC-specific device drivers. Closely related to this line of work is the new
ability to run multicore virtual machines on ARM.
Another highlight of version 20.11 is a new CPU balancing mechanism, which
automates the dynamic assignment of threads to CPU cores for complex
workloads. With traditional operating systems, such policies are normally part
of the OS kernel. Thanks to Genode's component architecture, we are able to
implement such potentially complex policies in the form of an optional
component, which offers ultimate flexibility while keeping the kernel
untainted by complex heuristics.
Further topics of the current release are improved power management and audio
support on PC hardware, a new OSS API emulation that allows for the
reuse of popular audio applications on Genode, and new support for VirtIO
networking. The full picture is given by the
[https:/documentation/release-notes/20.11 - release documentation of version 20.11...]
Sculpt OS release 20.08 | 2020-09-17
####################################
| Version 20.08 of the Sculpt operating system refines the user experience
| and becomes able to host the Chromium-based Falkon web browser.
The new version of Sculpt OS is based on the latest Genode release
[https://genode.org/documentation/release-notes/20.08 - 20.08].
In particular, it incorporates the
[https://genode.org/documentation/release-notes/20.08#The_GUI_stack__restacked - redesigned GUI stack]
to the benefit of quicker boot times, improved interactive responsiveness, and
better pixel output quality.
It also removes the last traces of the
[https://genode.org/documentation/release-notes/20.05#Retired_Noux_runtime_environment - noux runtime].
Fortunately, these massive under-the-hood changes do not disrupt the
user-visible surface of Sculpt. Most users will feel right at home.
Upon closer inspection, there are couple of new features to appreciate. The
CPU-affinity of each component can now be restricted interactively by the user,
components can be easily restarted via a click on a button, font-size changes
have an immediate effect now, and the VESA driver (used when running Sculpt in
a virtual machine) can dynamically change the screen resolution.
Thanks to our continuous efforts of strengthening of the base system, Sculpt
OS has become able to host a first version of the Chromium-based Falkon web
browser from an installable package. Even though this version is still rough
around the edges and unoptimized, it already enables Sculpt users to casually
browse the modern web without the need for a virtual machine.
The updated [https://genode.org/documentation/articles/sculpt-20-08 - manual]
goes into detail about the use of the new system.
The ready-to-use system image for version 20.08 is available at the
[https://genode.org/download/sculpt - Sculpt download page].
Genode OS Framework release 20.08 | 2020-08-28
##############################################
| With Genode 20.08, the low-level GUI stack underwent a profound redesign,
| the Chromium web engine comes to life, the i.MX8 support covers clock and
| power management, and the CBE block encrypter becomes highly modular.
The most stunning feature of Genode 20.08 is most certainly the new ability
to host the Chromium web engine as native component in the form of the Falkon
web browser. This long-time project involved overcoming countless road
blocks along the way to the great benefit of Genode's users at large: building
bridges between 3rd-party build systems and Genode, covering seemingly obscure
corner cases of POSIX, solving instruction-cache invalidation issues on ARM -
just to name a few.
Under the hood, we took the release cycle as opportunity to tackle a major
surgery of the low-level GUI stack that we planned for more than two years.
The architectural change lays the groundwork for swapping out graphics and
input drivers on the fly without reboot. It also paves the ground for features
like screen capturing and remote desktop scenarios in a privacy-protecting
way.
On our mission of bringing the driver support for the 64-bit ARM-based i.MX8
SoC on par with our driver coverage on Intel PCs, the release introduces a
platform driver specifically for this SoC that covers clock and power
management. One step closer to using Sculpt OS on the MNT Reform laptop.
Furthermore, Genode's custom block encrypter called CBE received continuous
development. In particular, the cryptographic algorithm and trust anchor
have become pluggable modules. This will allow for tailoring the CBE to
custom products - like hardware trust anchors - without changing its
implementation.
These and many more improvements are covered in detail in the
[https:/documentation/release-notes/20.08 - release documentation of version 20.08...]
Genode OS Framework release 20.05 | 2020-05-28
##############################################
| Version 20.05 wraps up the consolidation of Noux with the C runtime,
| advances the device-driver infrastructure and virtualization support on ARM,
| reaches feature-completion of our block encryptor, and leverages seccomp for
| sandboxing components on top of Linux.
It has become almost a tradition to dedicate the spring release to topics
under the hood of the Genode OS framework, taking the time for careful
consolidation, and architectural improvements.
On the latter account, the Linux version gained an architectural revamp of its
inter-component communication model, leveraging the combination of the
seccomp and epoll kernel mechanisms to enable Genode's capability-based
security concept on this kernel. The new version applies strict sandboxing
to each component individually. In particular, the host file system is
shielded from Genode components and Genode's least-privilege access-control
scheme comes into effect.
As the second prominent architectural topic, the release features new
device-driver infrastructure for the ARM universe along with a unified
version of the formerly distinct virtual machine monitors for ARMv7 and
ARMv8. These are important steps to bring Genode on ARM to parity with the
x86 version.
Further highlights of the release are the first feature-complete version
of our custom block-encryption component, the improved management of CPU
affinities on x86, and new tracing utilities. Last but not least, the
release is accompanied with an updated version of the Genode Foundations
book, reflecting the changes of the framework since one year ago.
Read the details of the new release in the
[https:/documentation/release-notes/20.05 - release documentation of version 20.05...]
Sculpt OS release 20.02 | 2020-03-10
####################################
| Version 20.02 of the Sculpt operating system revisits the administrative
| user interface for a more intuitive and logical user experience.
With the release of Sculpt version 20.02, we follow our
[https://genode.org/about/road-map - roadmap's] mission to make Sculpt OS
easier to approach. In particular, we
[https://genodians.org/nfeske/2020-01-06-pending-sculpt-ui - identified] the
reliance on a command-line interface as a potential barrier of entry. As
Sculpt OS is not a Unix-like system, it should not require any Unix know-how
from the user. To relieve users from this burden, Sculpt 20.02 introduces
a custom graphical file browser and editor that can be used for interactively
inspecting and tweaking the state of the system. The traditional command-line
interface is still present as a fallback for advanced tasks though.
The updated [https://genode.org/documentation/articles/sculpt-20-02 - manual]
goes into detail about the use of the new system.
Thanks to the work of seasoned Genode developers, many software packages are
already available for the new version. These include virtual machine monitors
like VirtualBox, performance-monitoring tools, GUI components, Genode's custom
Unix runtime, and several applications and games. In particular, the software
depots offered by alex-ab, cnuke, skalk are worth exploring.
The ready-to-use system image for version 20.02 can be obtained from the
[https://genode.org/download/sculpt - Sculpt download page] and is
accompanied by matching
[https://genode.org/documentation/articles/sculpt-20-02 - documentation].
Genode OS Framework release 20.02 | 2020-02-28
##############################################
| With version 20.02, Genode makes Sculpt OS fit for running on i.MX 64-bit
| ARM hardware, optimizes the performance throughout the entire software stack,
| and takes the next evolutionary step of the user-facing side of Sculpt OS.
Without any doubt, Sculpt OS has been the driving motivation behind most
working topics featured by the new release. One particularly exciting line
of work is the enabling of Sculpt on i.MX-based 64-bit ARM hardware, which
touched the framework on all levels, from the boot loader, over the kernel,
device drivers, libraries, system management, up to the application level.
The work goes as far as supporting Sculpt OS as a hypervisor platform for
hosting Linux in a virtual machine.
As a second Sculpt-related development, we strive to make the user-visible
side of the operating system better approachable and more logical. With this
background, the current release comes with a profound redesign of the
administrative user interface of Sculpt OS. An updated downloadable system
image will follow soon.
Also related to Sculpt are an updated audio driver based on OpenBSD 6.6,
the support of virtual desktops, and performance optimization of the
Seoul virtual machine monitor on x86 hardware.
Regarding the framework API, the release introduces a new library for
building multi-component applications. It aims to bring the benefits of
Genode's unique security architecture from the operating-system level to the
application level.
These topics are only the tip of the iceberg. For the complete picture,
please consult the
[https:/documentation/release-notes/20.02 - release documentation of version 20.02...]
Road Map for 2020 | 2020-01-20
##############################
| In 2020, we will be concerned about dwarfing the barrier of entry into
| the Genode world.
Following the last year's leitmotif of "bridging worlds", we turn our
attention to the removal of the hurdles faced by aspiring developers and
users. During the annual road-map
[https://lists.genode.org/pipermail/users/2019-December/006987.html - discussion]
on our mailing list, we identified four tangible approaches towards that
goal. First, making Sculpt OS more user friendly. Second, reinforcing trust in
Genode by fostering the framework's high quality. Third, making the tooling
around Genode a joy to use. And finally, the illustration of Genode's
versatility in the form practical use cases.
Besides this overall theme, we plan to continue our commitment to the
NXP i.MX SoC family, revisit Genode's low-latency audio support, and
extend the cultivation of Ada/SPARK within (and on top of) Genode.
More background information about the new road map and a rough schedule are
presented at our official [https:/about/road-map - road-map page].
Genode OS Framework release 19.11 | 2019-11-28
##############################################
| Following this year's theme of "bridging worlds", Genode 19.11 adds the
| ability to use popular build tools like CMake for application development,
| introduces a new virtual-machine monitor for 64-bit ARM, and enhances
| POSIX compatibility. As another highlight, it features the first version
| of our custom block-device encrypter.
Block-device encryption is a feature often requested by users of our Sculpt OS.
Until now, we deliberately left this topic unaddressed because we felt that a
profound answer was beyond our expertise. However, during the past year, we
dived deep into it. The result is the prototype for a new block encrypter that
encrypts data but also protects integrity and freshness. For us, the
implementation of the encrypter is especially intriguing because - with about
7000 lines of code - it is Genode's first non-trivial component written in the
[https://en.wikipedia.org/wiki/SPARK_(programming_language) - SPARK]
programming language.
The second major addition is a new virtual machine monitor (VMM) for 64-bit
ARM platforms such as the NXP i.MX8. It leverages the
[https://genode.org/documentation/articles/arm_virtualization - proof of concept]
we developed in 2015 for ARMv7, which we pursued as a technology exploration.
In contrast, our aspiration with the new VMM is a product-quality solution.
In our [https://genode.org/about/road-map - road map] for 2019, we stated
the "bridging of worlds" as our overall theme for this year. On that account,
the current release moves the project forward on two levels. First, by
successively increasing the scope of POSIX compatibility, we reduce the
friction when porting existing application software to Genode. We managed
to bridge several gaps in our POSIX support that we considered as impossible
to cover some years ago. In particular, we identified ways to emulate certain
POSIX signals, ioctl calls, and fork/execve semantics. This way, popular
software such as bash, coreutils, or Vim can now be executed as regular
Genode components with no additional runtime environment (like Noux or a VMM)
required.
At a higher level, the current release introduces new tooling especially
geared at the development and porting of application software. Compared to
Genode's regular development tools, which were designed for whole-system
development, the new tool called Goa relieves the developer from the
complexity of Genode's custom build system and instead promotes the use of
popular commodity solutions like CMake.
These and more topics are described in the
[https:/documentation/release-notes/19.11 - release documentation of version 19.11...]
Genode OS Framework release 19.08 | 2019-08-28
##############################################
| Genode 19.08 puts emphasis on practical concerns ranging from
| keyboard layouts, over system-time management, to remote system
| administration. It also continues our commitment to the 64-bit ARM
| i.MX8 SoC, comes with Qt5 version 5.13, and improves POSIX compatibility.
The summer release of Genode addresses a variety of topics when using Genode
and Sculpt OS in practice. The confrontation with the real world prompted us
to develop new concepts for managing system time, keyboards layouts, and
copy-and-paste. For using Sculpt OS on the road, a new application VM for
accessing captive portals smoothes the experience of connecting to public WiFi
networks.
Besides the practical focus, the new release continues our commitment to the
64-bit ARM i.MX8 platform through new kernel support, device drivers, and test
coverage. Further topics include SMBIOS support for commodity PC hardware, a
new tracing tool, enhanced POSIX compatibility, and a major update of Qt5 to
version 5.13.
The complete picture is presented in the
[https:/documentation/release-notes/19.08 - release documentation of version 19.08...]
Sculpt OS release 19.07 | 2019-07-09
####################################
| Version 19.07 of the Sculpt operating system improves overall performance
| and introduces copy and paste between terminals, virtual machines, and
| graphical applications.
The most prominent user-visible feature of Sculpt OS 19.07 is the ability
of copy and paste text between terminals, graphical applications, and
virtual machines. Our unique take on this feature is described in
a [https://genodians.org/nfeske/2019-07-03-copy-paste - dedicated article].
Under the hood, Sculpt 19.07 benefits from the massive infrastructure
improvements that came with
[https://genode.org/documentation/release-notes/19.05 - Genode 19.05],
yielding a smoother user experience compared to earlier versions.
The new release can be obtained from the
[https://genode.org/download/sculpt - Sculpt download page] and is
accompanied by updated
[https://genode.org/documentation/articles/sculpt-19-07 - documentation].
Genode OS Framework release 19.05 | 2019-05-29
##############################################
| The highlights of version 19.05 are a new kernel-agnostic virtualization
| interface, initial support for the 64-bit ARM architecture, the use of
| C++17 by default, a new tool chain based on GCC 8.3, updated C and SPARK
| runtimes, and the consolidation of build directories across boards.
We dedicated the release cycle of Genode 19.05 to platform topics at various
levels. The flagship feature is certainly the introduction of our
kernel-agnostic virtualization interface. It has been in the works for more
than a half year and gives us the prospect of running virtual machine monitors
like Seoul and VirtualBox seamlessly across Genode's supported kernels.
The second major theme is the extension of Genode's CPU-architecture support
to 64-bit ARM (AARCH64). This step motivated the update of many parts of the
framework's fundamental infrastructure, ranging from the tool chain (updated
to GCC 8.3), over the C runtime (updated to FreeBSD 12 libc), to the dynamic
linker. The new tool chain, in turn, paved the ground for enabling C++17 by
default.
With the diversity of kernels, CPU architectures, and boards growing, we are
constantly striving to remove friction and redundancies between Genode's
underlying platforms. The current release eventually consolidates the build
directories not only across kernels but also across all boards of a given
CPU architecture. This vastly increases the velocity of Genode-based system
scenarios when targeting multiple boards or emulators at the same time.
Further details about these and many more improvements are given in the
[https:/documentation/release-notes/19.05 - release documentation of version 19.05...]
Sculpt as a Community Experience | 2019-03-19
#############################################
| The fourth stage of Sculpt OS introduces a new federated software
| provisioning model while giving the user full control over the
| component deployment via a novel graphical user interface.
With Sculpt CE, we enter the final stage of the evolution of Sculpt OS as
envisioned roughly one year ago. Initially geared towards die-hard enthusiasts
only, each revision became more and more user friendly. The previous version
Sculpt VC already offered a glimpse of Sculpt's unique user interface in the
form of an interactive component graph.
The just released Sculpt OS "as a community experience" (CE) combines this
tangible notion of component compositions with a completely federated software
provisioning model that cuts out middlemen like an app store or a
distribution. With Sculpt CE, components can be offered by a federation of
independent software providers selectable by the user. The software
installation is sandboxed and protected via digital signatures. The
integration of components with the rest of the system is completely under
control by the user. With the principle of least privilege at the heart
of Sculpt's architecture, you - the user - can fearlessly install and run
software without the need to ultimately trust the software providers.
Sculpt CE is intended to work in tandem with the new community blog
[https://genodians.org - Genodians.org] where developers and users exchange
experiences and announce new software. The best way to watch how the
Sculpt story continues is the RSS feed of Genodians.org.
To dive into the new world of Sculpt CE,
[https://genode.org/download/sculpt - download Sculpt OS...]
Genodians.org | 2019-03-08
##########################
| Genodians.org is the new place to be for getting the latest news and stories
| around Genode. It is a federated blog by and for developers and users
| alike.
With [https://genodians.org - Genodians.org], the Genode community has gained
a new place for exchanging ideas, announcing current developments, giving
tutorials, and sharing experience stories. In contrast to the formal character
of Genode's regular release notes, the articles at Genodians.org are raw and
personal, authored by individuals with no editorial process. For feedback
about the articles, readers are invited to the new
[https://reddit.com/r/genode - /r/genode] subreddit.
As written in the
[https://genodians.org/nfeske/2019-01-07-welcome - initial posting],
Genode users and developers are warmly invited to join the authors at
Genodians.org!
As a side note, the blogging platform is based on the Genode OS framework and
is of course open source
([https://github.com/genodelabs/genodians.org - GitHub repository]).
[https://genodians.org - Visit Genodians.org...]
Genode OS Framework release 19.02 | 2019-02-28
##############################################
@@ -799,7 +1418,7 @@ The story behind Genode's TrustZone demo on the USB Armory | 2015-12-08
| Our latest article provides a look behind the scenes of the
| development of Genode's support for the USB Armory platform.
The [http://inversepath.com/usbarmory - USB Armory] is a computer in the form
The [https://inversepath.com/usbarmory - USB Armory] is a computer in the form
of a USB stick. It normally runs Linux. But thanks to the ARM TrustZone
capabilities of the device, it is possible to run Genode behind the back of
Linux. This is useful for shielding sensitive information like cryptographic
@@ -813,7 +1432,7 @@ was splitting the hardware platform into two worlds while maintaining the
full functionality of Linux. The article goes on to explain the interplay
between the secure world (Genode) and the normal world (Linux). Furthermore,
it provides all the pointers needed to reproduce the scenario.
[http:/documentation/articles/usb_armory - Read the article...]
[https://genode.org/documentation/articles/usb_armory - Read the article...]
Genode OS Framework release 15.11 | 2015-11-30
@@ -1992,7 +2611,7 @@ for our dual-licensing business model. If we made Genode available under the
BSD license, there would be not point in pursuing this model. However, hiding
the development process from the public is not only poor-spirited but it
creates an artificial barrier for people who want to participate. The book
"Producing Open Source Software" (http://producingoss.com) by Karl Fogel was an
"Producing Open Source Software" (https://producingoss.com) by Karl Fogel was an
eye opener to us.
Regarding the efficiency of collaboration, I have to admit that the statement
@@ -2090,7 +2709,7 @@ new support of L4Android on Genode, Android can be used on Genode on the
Fiasco.OC kernel on the IA32 architecture. For those of you who are eager to
experiment with L4Android on Genode, please find further information at the
top-level
[http://genode.svn.sourceforge.net/viewvc/genode/trunk/ports-foc/README - README]
[https://genode.svn.sourceforge.net/viewvc/genode/trunk/ports-foc/README - README]
file of the 'ports-foc' repository and share your results with us at the
[https:/community/mailing-lists - Genode mailing list].
@@ -2205,17 +2824,17 @@ concepts live. The talk was recorded at the
Amsterdam. Thanks to Bas the Lange for publishing the material.
: <object height="385" width="480">
: <param name="movie" value="http://www.youtube.com/v/Z1IMV3FJO7Q" />
: <param name="movie" value="https://www.youtube.com/v/Z1IMV3FJO7Q" />
: <param name="allowFullScreen" value="true" />
: <param name="allowscriptaccess" value="always" />
: <embed width="480" height="385" allowfullscreen="true"
: allowscriptaccess="always"
: type="application/x-shockwave-flash"
: src="http://www.youtube.com/v/Z1IMV3FJO7Q">
: src="https://www.youtube.com/v/Z1IMV3FJO7Q">
: </embed>
: </object>
[http://www.youtube.com/watch?v=Z1IMV3FJO7Q - Visit the YouTube page...]
[https://www.youtube.com/watch?v=Z1IMV3FJO7Q - Visit the YouTube page...]
Genode OS Framework release 11.02 | 2011-02-24
@@ -2272,17 +2891,17 @@ prepared the following screencast with a guided walk-through. Enjoy!
: <object height="385" width="480">
: <param name="movie"
: value="http://www.youtube.com/v/CJdWOmajo_8?fs=1&amp;hl=en_US" />
: value="https://www.youtube.com/v/CJdWOmajo_8?fs=1&amp;hl=en_US" />
: <param name="allowFullScreen" value="true" />
: <param name="allowscriptaccess" value="always" />
: <embed width="480" height="385"
: src="http://www.youtube.com/v/CJdWOmajo_8?fs=1&amp;hl=en_US"
: src="https://www.youtube.com/v/CJdWOmajo_8?fs=1&amp;hl=en_US"
: type="application/x-shockwave-flash" allowscriptaccess="always"
: allowfullscreen="true">
: </embed>
: </object>
[http://www.youtube.com/watch?v=CJdWOmajo_8 - Visit the YouTube page...]
[https://www.youtube.com/watch?v=CJdWOmajo_8 - Visit the YouTube page...]
[https:/download/live-cds - Download the real thing...]
@@ -2538,7 +3157,7 @@ applications, most prominently, it serves as the foundation of the KDE project.
Since the release 9.05, the official distribution of Genode supports Qt4 as a
regular feature. The document "Portierung von Qt auf Genode" _(german)_
describes the challenging endeavor of porting this high-complexity C++
framework to Genode. Major problems to overcome had been the missing C libary
framework to Genode. Major problems to overcome had been the missing C library
(at the time when the project started), the integration of the Qt4 project
files with Genode's build system, the adaption of Qt4 to the basic primitives
provided by Genode, and the integration of Qt4 with Genode's GUI. In addition
@@ -2681,13 +3300,13 @@ for graphics, input devices, and sound. It is often used as back end for games,
emulators, and media players. Also the Linux version of Genode relies on the
hardware abstractions provided by libSDL.
As [http://sourceforge.net/mailarchive/message.php?msg_id=21406424 - announced on the Genode mailing list],
As [https://sourceforge.net/mailarchive/message.php?msg_id=21406424 - announced on the Genode mailing list],
libSDL has been ported to Genode. At the current stage, the port supports
the video subsystem and the input handling for mouse and keyboard. With
libSDL now becoming available for Genode, it becomes much easier to make the
wealth of libSDL-based applications available on our platform.
[http://sourceforge.net/mailarchive/message.php?msg_id=21406424 - Read the announcement...]
[https://sourceforge.net/mailarchive/message.php?msg_id=21406424 - Read the announcement...]
Genode on the L4ka::Pistachio kernel | 2008-12-18
@@ -2839,7 +3458,7 @@ Project website launched | 2008-07-29
| Genode OS framework is scheduled for the 6th of August.
Today, we proudly launched the website of the Genode project
[https://www.genode.org - https://www.genode.org]. This website is the central
[https://genode.org - https://genode.org]. This website is the central
resource for people using or developing the Genode OS framework. It covers
the latest news about our progress, architectural and technical documentation,
a community-maintained wiki, mailing lists, information on accessing the
@@ -2848,7 +3467,7 @@ attention of people who want bring forward the project together with us.
We have scheduled the first official release of the Genode OS framework for the
6th August. Until then, we invite you to test-drive the beta-version of the
framework as provided at the [https://www.genode.org/download - download].
framework as provided at the [https://genode.org/download - download].
Genode Labs founded | 2008-07-17

830
doc/release_notes-08-11.txt
View File

@@ -1,830 +0,0 @@
==============================================
Release notes for the Genode OS Framework 8.11
==============================================
Genode Labs
Summary
#######
This document presents the new features and major changes introduced
in version 8.11 of the Genode OS Framework. It is geared towards
people interested in closely following the progress of the Genode
project and to developers who want to adopt their software to our
mainline development. The document aggregates important fragments
of the updated documentation such that you won't need to scan existing
documents for the new bits. Furthermore, it attempts to provide our
rationale behind the taken design decisions.
The general theme for the release 8.11 is enabling the use of the
Genode OS framework for real-world applications. Because we regard
the presence of device drivers and a way to reuse existing library
code as fundamental prerequisites for achieving this goal, the major
new additions are an API for device drivers written in C, an API for
handling asynchronous notifications, and a C runtime. Other noteworthy
improvements are the typification of capabilities at the C++-language
level, a way for receiving and handling application faults, the
introduction of managed dataspaces, and a new API for scheduling
timed events.
Base framework
##############
This section documents the new features and changes affecting the
'base' repository, in particular the base API.
New features
============
Connection handling
~~~~~~~~~~~~~~~~~~~
The interaction of a client with a server involves the definition of
session-construction arguments, the request of the session creation via
its parent, the initialization of the matching RPC-client stub code
with the received session capability, the actual use of the session
interface, and the closure of the session. A typical procedure of
using a service looks like this:
!#include <rom_session/client.h>
!...
!
!/* construct session-argument string and create session */
!char *args = "filename=config, ram_quota=4K");
!Capability session_cap = env()->parent()->session("ROM", args);
!
!/* initialize RPC stub code */
!Rom_session_client rsc(session_cap);
!
!/* invoke remote procedures, 'dataspace' is a RPC function */
!Capability ds_csp = rsc.dataspace();
!...
!
!/* call parent to close the session */
!env()->parent()->close(session_cap);
Even though this procedure does not seem to be overly complicated,
is has raised the following questions and criticism:
* The quota-donation argument is specific for each server. Most services
use client-donated RAM quota only for holding little meta data and,
thus, are happy with a donation of 4KB. Other services maintain larger
client-specific state and require higher RAM-quota donations. The
developer of a client has to be aware about the quota requirements for
each service used by his application.
* There exists no formalism for documenting session arguments.
* Because session arguments are passed to the 'session'-call as a plain
string, there are no syntax checks for the assembled string performed
at compile time. For example, a missing comma would go undetected until
a runtime test is performed.
* There are multiple lines of client code needed to open a session to
a service and the session capability must be maintained manually for
closing the session later on.
The new 'Connection' template provides a way to greatly simplify the
handling of session arguments, session creation, and destruction on the
client side. By implementing a service-specific connection class
inherited from 'Connection', session arguments become plain constructor
arguments, session functions can be called directly on the 'Connection'
object, and the session gets properly closed when destructing the
'Connection'. By convention, the 'Connection' class corresponding to a
service resides in a file called 'connection.h' in the directory of the
service's RPC interface. For each service, a corresponding 'Connection'
class becomes the natural place where session arguments and quota
donations are documented. With this new mechanism in place, the example
above becomes as simple as:
!#include <rom_session/connection.h>
!...
!
!/* create connection to the ROM service */
!Rom_connection rom("config");
!
!/* invoke remote procedure */
!Capability ds_csp = rom.dataspace();
[http://genode.org/documentation/api/base_index#Connecting_to_services - See the API documentation for the connection template...]
Typed capabilities
~~~~~~~~~~~~~~~~~~
A plain 'Capability' is an untyped reference to a remote object of any
type. For example, a capability can reference a thread object or a
session to a service. It is loosely similar to a C void pointer, for which
the programmer maintains the knowledge about which data type is actually
referenced. To facilitate the type-safe use of RPC interfaces at the C++
language level, we introduced a template for creating specialized
capability types ('Typed_capability' in 'base/typed_capability.h') and
the convention that each RPC interface declares a dedicated capability
type. Note that type-safety is not maintained across RPC interfaces. As
illustrated in Figure [layered_ipc], typification is done at the
object-framework level on the server side and via in the 'Connection'
classes at the client side.
[image layered_ipc]
From the application-developer's perspective, working with capabilities
has now become type-safe, making the produced code more readable and robust.
[http://genode.org/documentation/api/base_index#Capability_representation - See the updated API documentation for the capability representation...]
Fifo data structure
~~~~~~~~~~~~~~~~~~~
Because the 'List' data type inserts new list elements at the list head,
it cannot be used for implementing wait queues requiring first-in
first-out semantics. For such use cases, we introduced a dedicated
'Fifo' template. The main motivation for introducing 'Fifo' into the
base API is the new semaphore described below.
[http://genode.org/documentation/api/base_index#Structured_data_types - See the new API documentation for the fifo template...]
Semaphore
~~~~~~~~~
Alongside lock-based mutual exclusion of entering critical sections,
organizing threads in a producer-consumer relationship via a semaphore
is a common design pattern for thread synchronization. Prior versions
of Genode provided a preliminary semaphore implementation as part of
the 'os' repository. This implementation, however, supported only one
consumer thread (caller of the semaphore's 'down' function). We have
now enhanced our implementation to support multiple consumer threads
and added the semaphore to Genode's official base API. We have made
the wake-up policy in the presence of multiple consumers configurable
via a template argument. The default policy is first-in-first-out.
[http://genode.org/documentation/api/base_index#Synchronization - See the new API documentation for the semaphore...]
Thanks to Christian Prochaska for his valuable contributions to the new
semaphore design.
Asynchronous notifications
~~~~~~~~~~~~~~~~~~~~~~~~~~
Inter-process communication via remote procedure calls requires both
communication partners to operate in a synchronous fashion. The caller
of an RPC blocks as long as the RPC is not answered by the called
server. In order to receive the call, the server has to explicitly
wait for incoming messages. There are a number of situations where
synchronous communication is not suited.
For example, a GUI server wants to deliver a notification to one of its
clients about new input events being available. It does not want to
block on a RPC to one specific client because it has work to do for
other clients. Instead, the GUI server wants to deliver this
_notification_ with _fire-and-forget_ semantics and continue with
its operation immediately, regardless of whether the client received
the notification or not. The client, in turn, does not want to poll
for new input events at the GUI server but it wants to be _waken_up_
when something interesting happens. Another example is a block-device
driver that accepts many requests for read/write operations at once.
The operations may be processed out of order and may take a long time.
When having only synchronous communication available, the client and
the block device driver would have to employ one distinct thread for
each request, which is complicated and a waste of resources. Instead,
the block device driver just wants to acknowledge the completeness of
an operation _asynchronously_.
Because there are many more use cases for asynchronous inter-process
communication, we introduced a new signalling framework that complements
the existing synchronous RPC mode of communication with an interface for
issuing and receiving asynchronous notifications. It defines interfaces
for signal transmitters and signal receivers. A signal receiver can
receive signals from multiple sources, whereas the sources of incoming
signals are clearly distinguishable. One or multiple threads can either
poll or block for incoming signals. Each signal receiver is addressable
via a capability. The signal transmitter provides fire-and-forget
semantics for submitting signals to exactly one signal receiver. Signals
are communicated in a reliable fashion, which means that the exact number
of signals submitted to a signal transmitter is communicated to the
corresponding signal receiver. If notifications are generated at a higher
rate than as they can be processed at the receiver, the transmitter
counts the notifications and delivers the total amount with the next
signal transmission. This way, the total number of notifications gets
properly communicated to the receiver even if the receiver is not highly
responsive. Notifications do not carry any payload because this payload
would have to be queued at the transmitter.
[image signals]
Image [signals] illustrates the roles of signaller thread,
transmitter, receiver, and signal-handler thread.
[http://genode.org/documentation/api/base_index#Asynchronous_notifications - See the new API documentation for asynchronous notifications...]
The current generic implementation of the signalling API employs one
thread at each transmitter and one thread at each receiver. Because
the used threads are pretty heavy weight with regard to resource usage,
ports of Genode should replace this implementation with platform-
specific variants, for example by using inter-process semaphores or
native kernel support for signals.
Region-manager faults
~~~~~~~~~~~~~~~~~~~~~
In Genode, region-manager (RM) sessions are used to manage the
address-space layout for processes. A RM session is an address-space
layout that can be populated by attaching (portions of) dataspaces to
(regions of) the RM session. Normally, the RM session of a process is
first configured by the parent when decoding the process' ELF binary.
During the lifetime of the process, the process itself may attach
further dataspaces to its RM session to access the dataspace's content.
Core as the provider of the RM service uses this information for
resolving page faults raised by the process. In prior versions of
Genode, core ignored unresolvable page faults, printed a debug message
and halted the page-faulted thread. However, this condition may be of
interest, in particular to the process' parent for reacting on the
condition of a crashed child process. Therefore, we enhanced the RM
interface by a fault-handling mechanism. For each RM session, a fault
handler can be installed by registering a signal receiver capability.
If an unresolvable page fault occurs, core delivers a signal to the
registered fault handler. The fault handler, in turn, can request the
actual state of the RM session (page-fault address) and react upon
the fault. One possible reaction is attaching a new dataspace at the
fault address and thereby implicitly resolving the fault. If core
detects that a fault is resolved this way, it resumes the operation
of the faulted thread.
This mechanism works analogously to how page faults are handled by
CPUs, but on a more abstract level. A (n-level) page table corresponds
to a RM session, a page-table entry corresponds to a dataspace-
attachment, the RM-fault handler corresponds to a page-fault
exception handler, and the resolution of page-faults (RM fault)
follows the same basic scheme:
# Application accesses memory address with no valid page-table-entry
(RM fault)
# CPU generates page-fault exception (core delivers signal to fault
handler)
# Kernel reads exception-stack frame or special register to determine
fault address (RM-fault handler reads RM state)
# Kernel adds a valid page-table entry and returns from exception
(RM-fault handler attaches dataspace to RM session, core resumes
faulted thread)
The RM-fault mechanism is not only useful for detecting crashing child
processes but it enables a straight-forward implementation of growing
stacks and heap transparently for a child process. An example for
using RM-faults is provided at 'base/src/test/rm_fault'.
Note that this mechanism is only available on platforms on which core
resolves page faults. This is the case for kernels of the L4 family.
On Linux however, the Linux kernel resolves page faults and suspends
processes performing unresolvable memory accesses (segmentation fault).
Managed dataspaces (experimental)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The RM-fault mechanism clears the way for an exciting new feature
of Genode 8.11 called managed dataspaces. In prior versions of Genode,
each dataspace referred to a contiguous area of physical memory (or
memory-mapped I/O) obtained by one of core's RAM, ROM, or IO_MEM
services, hence we call them physical dataspaces. We have now added
a second type of dataspaces called managed dataspaces. In contrast
to a physical dataspace, a managed dataspace is backed by the content
described by an RM session. In fact, each RM session can be used as
dataspace and can thereby be attached to other RM sessions.
Combined with the RM fault mechanism described above, managed
dataspaces enable a new realm of applications such as dataspaces
entirely managed by user-level services, copy-on-write dataspaces,
non-contiguous large memory dataspaces that are immune to physical
memory fragmentation, process-local RM fault handlers (e.g., managing
the own thread-stack area as a sub-RM-session), and sparsely populated
dataspaces.
Current limitations
-------------------
Currently, managed dataspaces still have two major limitations. First,
this mechanism allows for creating cycles of RM sessions. Core must
detect such cycles during page-fault resolution. Although, a design for
an appropriate algorithm exists, cycle-detection is not yet implemented.
The missing cycle detection would enable a malicious process to force
core into an infinite loop. Second, RM faults are implemented using the
new signalling framework. With the current generic implementation, RM
sessions are far more resource-demanding than they should be. Once the
signalling framework is optimized for L4, RM sessions and thereby
managed dataspaces will become cheap. Until then, we do not recommend
to put this mechanism to heavy use.
Because of these current limitations, managed dataspaces are marked as
an experimental feature. When building Genode, experimental features are
disabled by default. To enable them, add a file called 'specs.conf'
with the following content to the 'etc/' subdirectory of your build
directory:
! SPECS += experimental
For an example of how to use the new mechanism to manage a part of a
process' own address space by itself, you may take a look at
'base/src/test/rm_nested'.
Changes
=======
Besides the addition of the new features described above, the following
parts of the base framework underwent changes worth describing.
Consistent use of typed capabilities and connection classes
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
We applied capability typification to all interfaces of Genode including
the base API and the interfaces defined in the 'os' repository. Figure
[base_cap_types] provides an overview about the capability types
provided by the base API.
[image base_cap_types]
Overview about the capability types provided by the base API
Furthermore, we have complemented all session interfaces with
appropriate 'Connection' classes taking service-specific session
arguments into account.
For session-interface classes, we introduced the convention to declare
the service name as part of the session-interface via a static member
function:
! static const char *service_name();
Allocator refinements
~~~~~~~~~~~~~~~~~~~~~
Throughout Genode, allocators are not only used for allocating memory
but also for managing address-space layouts and ranges of physical
resources such as I/O-port ranges or IRQ ranges. In these cases, the
address '0' may be a valid value. Consequently, this value cannot be
used to signal allocation errors as done in prior versions of Genode.
Furthermore, because managed dataspaces use the RM session interface to
define the dataspace layout, the address-'0' problem applies here as
well. We have now refined our allocator interfaces and the RM-session
interface to make them fit better for problems other than managing
virtual memory.
Misc changes
~~~~~~~~~~~~
We revised all interfaces to consistently use _exceptions_ to signal
error conditions rather than delivering error codes as return values.
This way, error codes become exception types that have a meaningful
name and, in contrast to global 'errno' definitions, an error exception
type can be defined local to the interface it applies to. Furthermore,
the use of exceptions allows for creating much cleaner looking interfaces.
Traditionally, we have provided our custom _printf_ implementation as C
symbol to make this function available from both C and C++ code. However,
we observed that we never called this function from C code and that the
'printf' symbol conflicts with the libc. Hence, we turned 'printf'
into a C++ symbol residing in the 'Genode' namespace.
Operating-system services and libraries
#######################################
This section documents the new features and changes affecting
the 'os' repository.
New Features
============
Device-driver framework for C device drivers
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Genode's base API features everything needed to create user-level device
drivers. For example, the 'os' repository's PS/2 input driver and the
PCI bus driver are using Genode's C++ base API directly. However, most of
today's device drivers are written in C. To ease the reuse of existing
drivers on Genode, we have introduced a C API for device drivers into
Genode's 'os' repository. The API is called DDE kit (DDE is an acronym
for device-driver environment) and it is located at 'os/include/dde_kit'.
The DDE kit API is the result of long-year experiences with porting device
drivers from Linux and FreeBSD to custom OS environments. The following
references are the most significant contributions to the development of
the API.
;
Christian Helmuth created the initial version of the Linux device-driver
environment for L4. He describes his effort of reusing unmodified sound
drivers on the L4 platform in his thesis
[http://os.inf.tu-dresden.de/papers_ps/helmuth-diplom.pdf - Generische Portierung von Linux-Gerätetreibern auf die DROPS-Architektur].
;
Gerd Griessbach approached the problem of re-using Linux USB drivers
by following the DDE approach in his diploma thesis
[http://os.inf.tu-dresden.de/papers_ps/griessbach-diplom.pdf - USB for DROPS].
;
Marek Menzer adapted Linux DDE to Linux 2.6 and explored the DDE
approach for block-device drivers in his student research project
[http://os.inf.tu-dresden.de/papers_ps/menzer-beleg.pdf - Portierung des DROPS Device Driver Environment (DDE) für Linux 2.6 am Beispiel des IDE-Treibers ]
and his diploma thesis
[http://os.inf.tu-dresden.de/papers_ps/menzer-diplom.pdf - Entwicklung eines Blockgeräte-Frameworks für DROPS].
;
Thomas Friebel generalized the DDE approach and introduced the DDE kit
API to enable the re-use of device driver from other platforms than
Linux. In particular, he experimented with the block-device drivers of
FreeBSD in his diploma thesis
[http://os.inf.tu-dresden.de/papers_ps/friebel-diplom.pdf - Übertragung des Device-Driver-Environment-Ansatzes auf Subsysteme des BSD-Betriebssystemkerns].
;
Dirk Vogt successfully re-approached the port of USB device drivers
from the Linux kernel to L4 in his student research project
[http://os.inf.tu-dresden.de/papers_ps/beleg-vogt.pdf - USB for the L4 Environment].
The current incarnation of the DDE kit API provides the following
features:
* General infrastructure such as init calls, assertions, debug output
* Interrupt handling (attach, detach, disable, enable)
* Locks, semaphores
* Memory management (slabs, malloc)
* PCI access (find device, access device config space)
* Virtual page tables (translation between physical and virtual
addresses)
* Memory-mapped I/O, port I/O
* Multi-threading (create, exit, thread-local storage, sleep)
* Timers, jiffies
For Genode, we have created a complete reimplementation of the DDE kit
API from scratch by fully utilizing the existing Genode infrastructure
such as the available structured data types, core's I/O services,
the synchronization primitives, and the thread API.
[image dde_kit]
Figure [dde_kit] illustrates the role of DDE kit when re-using an
unmodified device driver taken from the Linux kernel. DDE kit translates
Genode's C++ base API to the DDE kit C API. The DDE kit API, in turn, is
used as back end by the Linux driver environment, which translates Linux
kernel interfaces to calls into DDE kit. With this translation in place,
an unmodified Linux device driver can be embedded into the Linux driver
environment. The device API is specific for a class of devices such as
NICs, block devices, or input devices. It can either be used directly as
a function interface by an application that is using the device driver
as a library, or it can be made accessible to external processes via an
RPC interface.
Limitations
-----------
The PCI sub system is not completely implemented, yet.
Alarm API providing a timed event scheduler
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The scheduling of timed events is a recurring pattern found in device
drivers, application frameworks such as Qt4 ('qeventdispatcher'), and
applications. Therefore, we have added a timed event scheduler to the
'os' repository. The new alarm API ('os/include/os/alarm.h') allows
for the scheduling of both one-shot alarms and periodic alarms.
Changes
=======
PS/2 input driver
~~~~~~~~~~~~~~~~~
The original PS/2 driver tried to switch the PS/2 keyboard to
scan-code set 2 and assumed that all modern keyboards support this
mode of operation. However, this assumption was wrong. We observed
that the legacy PS/2 support of some USB keyboards covers only the
emulated (xlate) scan-code set 1 mode. This is also case for the PS/2
emulation in VirtualBox. Therefore, we changed our PS/2 driver to
never touch the keyboard mode but to only detect the current mode
of operation. The driver has now to support both, scan-code set 1 and
scan-code set 2. This change comes along with a slightly more complex
state machine in the driver. Hence, we moved the state machine from
the IRQ handler to a distinct class and changed the control flow of
the driver to fetch only one value from the i8042 PS/2 controller
per received interrupt.
PCI bus driver
~~~~~~~~~~~~~~
Until now, Genode's PCI bus driver was only used for experimentation
purposes. With the forthcoming driver framework however, the PCI bus
driver will play a central role in the system. Therefore, we adapted
the interface of the PCI driver to these requirements. Specifically,
the scanning of the PCI bus can now be performed without constraining
the results by a specific vendor ID.
Nitpicker GUI server
~~~~~~~~~~~~~~~~~~~~
We improved the _output_latency_ of the Nitpicker GUI server by flushing
pixels eagerly and deferring the next periodically scheduled flush.
This change has a positive effect on the responsiveness of the GUI to
user input.
Misc changes
~~~~~~~~~~~~
Prior versions of the 'os' repository came with a custom 'os/include/base'
directory with interfaces extending the base API. To avoid confusion
between the 'base' repository and the 'os' repository, 'os'-local API
extensions are now located at 'os/include/os'. This way, the folder
prefix of include statements indicates well from which repository the
included header files comes from.
C runtime
#########
Most of existing libraries rely on the presence of a C library. For
making the reuse of this software on Genode possible, we have now
made a complete C library available for Genode. It comes as a separate
source-code repository called 'libc' and is based on the code of FreeBSD.
The original code is available at the official FreeBSD website.
:FreeBSD website:
[http://www.freebsd.org/developers/cvs.html]
Our libc port comprises the libraries 'gdtoa', 'gen', 'locale', 'stdio',
'stdlib', 'stdtime', 'string', and 'msun'. Currently, it supports the
x86 architecture. Support for other architectures is planned as future
addition. At the current stage, our back end is very basic and most of
its functions are dummy stubs. We used Christian Prochaska's forthcoming
Genode port of Qt4 as test case and successfully used the new libc as
foundation for building graphical Qt4 applications. We will further
extend the back end in correspondence to the growing feature set of the
Genode OS framework.
:Usage:
To use the libc in your application, just add 'libc' to the 'LIBS'
declaration in your build-description file. This declaration will make
the libc headers available for the include path of your target and link
the C library. When building, make sure that the 'libc' repository is
included in your build configuration ('etc/build.conf').
:Limitations:
The current version of the C library is not thread-safe. For most
string and math functions, this is not a problem (as these functions
do not modify global state) but be careful with using more complex
functions such as 'malloc' from multiple threads. Also, 'errno' may
become meaningless when calling libc functions from multiple threads.
We have left out the following files from the Genode port of the
FreeBSD libc: gdtoa 'strtodnrp.c' (gdtoa), 'getosreldate.c' (gen),
'strcoll.c', 'strxfrm.c', 'wcscoll.c', 'wcsxfrm.c' (string),
's_exp2l.c' ('msun').
The current back end is quite simplistic and it may help you to revisit
the current state of the implementation in the 'libc/src/lib/libc'
directory. If one of the functions in 'dummies.c' is called, you will
see the debug message:
! "<function-name> called, not yet implemented!"
However, some of the back-end function implemented in the other files
have dummy semantics but have to remain quiet because they are called
from low-level libc code.
Build infrastructure
####################
Build-directory creation tool
=============================
Because we think that each Genode developer benefits from knowing the
basics about the functioning of the build system, the manual creation of
build directories is described in Genode's getting-started document.
However, for regular developers, creating build directories becomes a
repetitive task. Hence, it should be automated. We have now added a
simple build-directory creation tool that creates pre-configured build
directories for some supported platforms. The tool is located at
'tool/builddir/create_builddir'. To print its usage information, just
execute the tool without arguments.
Improved linking of binary files
================================
For linking binary data, binary file have to be converted to object
files. Over the time, we have used different mechanisms for this
purpose. Originally, we used 'ld -r -b binary'. Unfortunately, these
linker options are not portable. Therefore, the mechanism was changed
to a 'hexdump' and 'sed' magic that generated a C array from binary data.
This solution however, is complicated and slow. Now, we have adopted
an idea of Ludwig Hähne to use the 'incbin' directive of the GNU
assembler, which is a very clean, flexible, and fast solution.
Lib-import mechanism
====================
Libraries often require specific include files to be available at the
default include search location. For example, users of a C library
expect 'stdio.h' to be available at the root of the include search
location. Placing the library's include files in the root of the
default search location would pollute the include name space for
all applications, regardless if they use the library or not. To
keep library-include files well separated from each other, we have
enhanced our build system by a new mechanism called lib-import.
For each library specified in the 'LIBS' declaration of a build
description file, the build system incorporates a corresponding
'import-<libname>.mk' file into the build process. Such as file
defines library-specific compiler options, in particular additional
include-search locations. The build system searches for lib-import
files in the 'lib/import/' subdirectories of all used repositories.
Using 'ar' for creating libraries
=================================
The previous versions of Genode relied on incremental linking ('ld -r')
for building libraries. This approach is convenient because the linker
resolves all cross-dependencies between libraries regardless of the
order of how libraries are specified at the linker's command line.
However, incremental linking prevents the linker from effectively
detecting dead code. In contrast, when linking '.a' files, the linker
detects unneeded object files. Traditionally, we have only linked our
own framework containing no dead code. This changed with the new 'libc'
support. When linking the 'libc', the presence of dead code becomes
the normal case rather than the exception. Consequently, our old
incremental-linking approach produced exceedingly large binaries
including all functions that come with the 'libc'. We have now adopted
the classic 'ar' mechanism for assembling libraries and use the linker's
'start-group' 'end-group' feature to resolve inter-library-dependencies.
This way, dead code gets eliminated at the granularity of object files.
In the future, we will possible look into the '-ffunction-sections' and
'-gc-sections' features of the GNU tool chain to further improve the
granularity to function level.
If your build-description files rely on custom rules referring to
'lib.o' files, these rules must be adapted to refer to 'lib.a' files
instead.
Misc changes
============
* Added sanity check for build-description files overriding 'INC_DIR'
instead of extending it.
* Restrict inclusion of dependency files to those that actually matter
when building libraries within 'var/libcache'. This change significantly
speeds up the build process in the presence of large libraries such as
Qt4 and libc.
* Added rule for building 'cpp' files analogously to the 'cc' rule.
Within Genode, we name all C++ implementation files with the 'cc'
suffix. However, Qt4 uses 'cpp' as file extension so we have to
support both.
* Build-description files do no longer need the declaration
'REQUIRES = genode'. Genode's include search locations are now
incorporated into the build process by default.
Applications
############
This section refers to the example applications contained in Genode's
'demo' repository.
We have enhanced the _Scout_widgets_ as used by the launchpad and the
Scout tutorial browser to perform all graphical output double-buffered,
which effectively eliminates drawing artifacts that could occur when
exposing intermediate drawing states via direct (unbuffered) output.
Furthermore, we have added a way to constrain the maximum size of
windows to perform pixel-buffer allocations on realistic window sizes.
Both launchpad and Scout can now start child applications. In Scout
this functionality is realized by special "execute" links. We have
generalized the underlying application logic for creating and
maintaining child processes between both applications and placed
the unification into a separate 'launchpad' library.
We have replaced the default document presented in Scout with an
_interactive_walk-through_guide_ explaining the basic features of Genode.
The document uses the new "execute" link facility to let the user start
a launchpad instance by clicking on a link.
Platform-specific changes
#########################
Genode used to define _fixed-width_integer_types_ in a file 'stdint.h'
placed in a directory corresponding to bit-width of the platform, for
example 'include/32bit/stdint.h'. When building for a 32bit platform,
the build system included the appropriate directory into the
include-search path and thereby made 'stdint.h' available at the root
of the include location. Unfortunately, this clashes with the 'stdint.h'
file that comes with the C library. To avoid conflict with libc header
files, we moved the definition of fixed-width integer types to
'32bit/base/fixed_stdint.h'.
For the L4/Fiasco version of Genode, there existed some x86-specific
header files that did not specifically depend on L4/Fiasco, for example
atomic operations. Because these files are not L4/Fiasco-specific and
may become handy for other platforms as well, we moved them to the
generic 'base' repository.
Linux 32bit
===========
:Dissolving Genode's dependency from the glibc:
The port of the C runtime to Genode posed an interesting challenge to
the Linux version of Genode. This version used to rely on certain
functions provided by the underlying glibc:
* For creating and destroying threads, we used to rely on POSIX threads
as provided by the 'pthread' library
* The lock implementation was based on the POSIX semaphore functions
'sem_init', 'sem_wait', and 'sem_post'
* Shared memory was realized by using files ('open', 'close',
'ftruncate') and the 'mmap' interface
* Starting and killing processes was implemented using 'fork' and 'kill'
* Inter-process communication used the glibc's socket functions
For our custom C runtime, we want to override the glibc functionality
with our own implementation. For example, we want to provide the 'mmap'
interface to a Genode application by implementing 'mmap' with
functions of our base API. On Linux, however, this base API, in turn,
used to rely on 'mmap'. This is just an example. The problem applies
also for the other categories mentioned above. We realized that we cannot
rely on the glibc on one hand but at the same time replace it by a custom
C runtime (in fact, we believe that such a thing is possible by using
awkward linker magic but we desire a clean solution). Consequently, we
have to remove the dependency of Genode from the glibc on Linux. Step
by step, we replaced the used glibc functions by custom Linux system-call
bindings. Each binding function has a prefix 'lx_' such that the symbol
won't collide with 'libc' symbols. The new bindings are located at the file
'base-linux/src/platform/linux_syscalls.h'. It consist of 20 functions,
most of them resembling the original interface ('socket', 'connect',
'bind', 'getsockname', 'recvfrom', 'write', 'close', 'open', 'fork',
'execve', 'mmap', 'ftruncate', 'unlink', 'tkill', 'nanosleep').
For other functions, we simplified the semantics for our use case
('sigaction', 'sigpending', 'sigsetmask', 'create_thread'). The most
noteworthy changes are the creation and destruction of threads by
directly using the 'clone' and 'tkill' system calls, and the lock
implementation. Because we cannot anymore rely on the convenience of
using futexes indirectly through the POSIX semaphore interface, we
have adopted the simple locking approach that we already use for the
L4/Fiasco version. This lock implementation is a simple sleeping
spinlock.
:Compromises:
The introduction of custom Linux system-call bindings for Genode has
several pros and cons. With this change, The Linux version of Genode is
not anymore easy to port to other POSIX platforms such as the Darwin
kernel. For each POSIX kernel used as Genode platform, a custom
implementation of our system-call bindings must be created. The
original POSIX variant could still be reanimated, but this version
would inherently lack support for Genode's C runtime, and thus would
have limited value. A positive side effect of this solution, however,
is that 'linux_syscalls.h' documents well the subset of the Linux'
kernel interface that we are actually using.
The replacement of POSIX semaphores with sleeping spinlocks decreases
locking performance quite significantly. In the contention case, the
wakeup from sleeping introduces a high latency of up to one millisecond.
Furthermore, fairness is not guaranteed and the spinning produces a bit
of system load. If this approach turns out to become a serious performance
bottleneck, we will consider creating custom bindings for Linux' futexes.
L4/Fiasco
=========
The concepts of _RM_faults_ and _managed_dataspaces_ as described in
Section [Base framework], had been implemented into the L4/Fiasco
version of core. Although the introduction of these concepts involved
only minimal changes at the API level, the required core-internal
changes had been quite invasive, affecting major parts of the pager
and RM-session implementations.
Prior versions of the L4/Fiasco version of core did not implement
the _cancel-blocking_mechanism_ as specified by the CPU-session API.
The missing implementation resulted in lock-ups when destructing a
thread that blocks for lock. With the new implementation based on
L4/Fiasco's inter-task ex-regs system call, such threads can now
be gracefully destructed.

460
doc/release_notes-09-02.txt
View File

@@ -1,460 +0,0 @@
==============================================
Release notes for the Genode OS Framework 9.02
==============================================
Genode Labs
Summary
#######
Whereas the focus of the previous release 8.11 was the refinement of
Genode's base API and the creation of the infrastructure needed to build
real-world applications, the release 9.02 is focused on functional
enhancements in two directions. The first direction is broadening the
number of possible base platforms for the framework. At present, most
microkernels bring along a custom user land, which is closely tied to the
particular kernel. Our vision is to establish Genode as a common ground for
developing applications, protocol stacks, and device drivers in such a way
that the software becomes easily portable among different kernels. This
release makes Genode available on the L4ka::Pistachio kernel. Hence,
software developed with the Genode API can now run unmodified on
Linux/x86, L4/Fiasco, and L4ka::Pistachio. In the second direction, we
are steadily advancing the functionality available on top of Genode. With
this release, we introduce a basic networking facility and support for
native Qt4 applications as major new features. Thanks to Genode's
portability, these features become automatically available on all
supported base platforms.
Our original plan for the release 9.02 also comprised the support of a
Linux-on-Genode (para-)virtualization solution. Initially, we intended to
make [http://os.inf.tu-dresden.de/L4/LinuxOnL4/ - L4Linux] available on
the L4/Fiasco version of Genode. However, we identified several downsides
with this approach. Apparently, the development of the officially available
version of L4/Fiasco has become slow and long-known issues remain unfixed.
L4Linux, however, is closely tied to L4/Fiasco and the L4 environment. For
us at Genode Labs, maintaining both a custom port of L4Linux for Genode
and L4/Fiasco by ourself in addition to developing Genode is unfeasible.
In contrast, the Pistachio kernel features more advanced options for
virtualization ([http://l4ka.org/projects/virtualization/afterburn/ - Afterburner]
and VT support) that we want to explore. Furthermore, there exists another
version of L4Linux called OKLinux for the OKL4 kernel developed at
[http://ok-labs.com - OK-Labs], which is very interesting as well.
Therefore, we decided against an ad-hoc solution and deferred this feature
to the next release. [http:/about/road-map - See our updated road map...]
Major new Features
##################
Genode on L4ka::Pistachio
=========================
From the very beginning, the base API of the Genode OS Framework was
designed for portability. We put a lot of effort into finding API
abstractions that are both implementable on a wide range of kernels and as
close to the hardware as possible to keep the abstraction overhead low. For
this reason, we developed the framework in parallel for the Linux kernel and
the L4/Fiasco kernel. To validate our claim that Genode is highly portable,
Julian Stecklina ported the framework to another member of the L4 family,
namely the [http://l4ka.org/projects/pistachio/ - L4ka::Pistachio kernel].
This high-performance kernel implements the latest official L4 API called
L4.x2 and has a number of advanced features such as multi-processor support
and virtualization support.
After Julian successfully created the first Pistachio version of Genode,
we successively refined his work and conducted further unifications among
the platform-dependent code for the different kernels. The result of this
effort is included in this release and comes in the form of the
'base-pistachio' source-code repository.
;Interesting technical notes:
* The IRQ handling on Pistachio is slightly different from L4/Fiasco.
On L4/Fiasco, an IRQ becomes unmasked only when the user-level IRQ
handler thread blocks for an IRQ by issuing an IPC call to the
kernel's corresponding IRQ thread. In contrast, Pistachio unmasks an
IRQ as soon as the user-level IRQ handler associates itself with an
IRQ. Thus, an IRQ message may occur not only when the user-level IRQ
handler blocks for any IRQ but anytime. In particular, IRQ messages
may interfere with the IRQ handler's IPC communication with other
threads. To ensure that IRQ messages do only occur when expecting
them, we lazily associate the IRQ handler thread to the IRQ the
first time we wait for an IRQ and issue an unmasking IPC call
subsequent times.
* Genode provides a mechanism for gracefully destructing threads that
are in a blocking state, for example waiting for an IPC message.
Such a thread may hold locks or other resources that would not
get properly freed when just killing the thread by force. Therefore,
Genode provides a way to issue the cancellation of a blocking
operation by another thread (e.g., the thread calling the destructor).
Once, a blocking operation got canceled, a C++ exception
('Blocking_canceled') is raised such the thread can fall back into
a defined state and then be destructed. On L4ka::Pistachio, we use
Pistachio's pager-exchange-registers feature in combination with
the user-defined UTCB handle for cancelling blocking operations and
detecting cancellations. The interesting code bits can be found in
'src/base/ipc/ipc.cc', 'src/base/lock/lock.cc',
'src/core/platform_thread.cc', and in the Pistachio-specific
timer-service implementation.
* During the refinement of the Pistachio version, we were able to further
generalize code that was previously specific for L4/Fiasco and
L4ka::Pistachio respectively. Now, the platform-specific code comprises
less than 3,000 lines of code (LOC) for L4/Pistachio, circa 2,000 LOC
for L4/Fiasco, and circa 1,000 LOC for Linux. Hence, we expect that
porting the framework to further kernels is possible at reasonable
engineering costs.
:Current limitations:
* The current version does not use superpages (4M mappings) because we
experienced problems with mapping 4K pages out of 4M pages. This is an
issue that we like to investigate further because using 4M mappings
would improve the boot time and reduce the kernel-memory usage.
* Currently, we use a simple sleeping spinlock for synchronization, which
is not optimal for several reasons. There are no fairness guarantees,
the spinning consumes CPU time, and threads that got blocked in the
contention case are woken up at the coarse granularity of the kernel's
timer tick, which is typically one millisecond.
* Nested RM sessions as introduced as an experimental feature in the
Genode release 8.11 are not yet supported.
:Further details:
You can find further technical details and usage instructions at this
dedicated [http://genode.org/documentation/platforms/pistachio - page].
Qt4 on Genode
=============
The minimalism of the Genode OS Framework with regard to its code
complexity raised the question of whether this framework is feasible
for hosting real-world applications and widely used runtime environments.
Christian Prochaska took the challenge to port Trolltech's Qt4 application
framework, which serves as the basis for the popular KDE desktop, to Genode.
Because Christian started his work more than a year ago at a time when no
C library was available on Genode, several intermediate steps were needed.
The first step was the integration of the Qt4 tools such as the meta-object
compiler (moc) and resource compiler properly into the our build systion.
With the tools in place, the Linux version of Genode came to an advantage.
In this environment, a Genode application is able to use glibc functionality.
So the problem of a missing C library could be deferred and Christian was
able to focus on interfacing Qt with the existing Genode services such as
the Nitpicker GUI sever. Next, the glibc dependencies were successively
replaced by custom implementations or simple dummy stubs. Thereby, all
needed functionalities such as timed semaphores and thread-local storage
had to be mapped to existing Genode API calls. Once, all glibc dependencies
had been dissolved, Qt could be compiled for the L4/Fiasco version.
Since a C library has become available in Genode 8.11, we were able to
replace Christian's intermediate stub codes with a real C library. We also
utilize recently added features of Genode such as its alarm framework to
simplify the Qt4 port. Furthermore, we were able to remove all
platform-specific bits such that the Qt4 port has now become completely
generic with regard to the underlying kernel. Qt4 can be executed on Linux,
L4/Fiasco, and L4ka::Pistachio without any changes. Figure [qt4_screenshot]
shows a screenshot of Qt's Tetrix example running side-by-side with native
Genode applications.
[image qt4_screenshot]
:Current state:
* The Qt4 port comes in the form of a source-code repository, which contains
all Qt source codes, and some example programs such as Tetrix. You can
download the Qt4 repository as a separate archive at the download page of
the Genode release 9.2. For the next release, we plan to separate the
Genode-specific parts from Qt original code and make the Genode-specific
parts a regular component of the Genode main line.
* The Qt4 port consists of Qt's Core library, GUI library, Script library,
XML library, and the UI tools library. Other libraries such as Webkit
are not ported yet.
* This first version of Qt4 on Genode is not to be considered as stable.
There are several known issues yet to be addressed. In particular,
the 'QEventDispatcher' is still work in progress and not fully stabilized.
* Because, we use to statically link programs, the binaries of Qt
applications are exceedingly large. For example the Tetrix binary is
100MB including debug information and 11MB in the stripped form. For
employing Qt on Genode at a larger scale, Genode should be enhanced with
shared-library support.
Networking
==========
With Genode 8.11, we introduced the Device-Driver-Environment Kit (DDE Kit)
API, which is a C API specifically designed for implementing and porting
device drivers written in plain C. We have now complemented DDE Kit with an
environment for executing Linux device drivers natively on Genode. This
library is called 'dde_linux26' and contained in our new 'linux_drivers'
source-code repository. The environment consists of several parts, which
correspond to the different sub systems of the Linux kernel 2.6, such as
'arch', 'drivers', 'kernel'.
The first class of device-drivers supported by DDE Linux 2.6 is networking.
At the current stage, the DDE Linux network library comprises general
network-device infrastructure as well as an exemplary driver for the PCnet32
network device.
Based on this library, we have created a basic TCP/IP test utilizing the
uIP stack, which uses the DDE Linux network library as back end. The test
program implements a basic web server displaying uIP packet statistics.
When executed on Qemu, you can use your host's web browser to connect to
the web server running on Genode:
For booting Genode on L4/Fiasco with the web-server demo, use a GRUB
entry in your 'menu.lst' file as follows.
! title Genode: DDE Linux 2.6 NET on L4/Fiasco
! kernel /fiasco/bootstrap -maxmem=64 -modaddr=0x02000000
! module /fiasco/fiasco -nokd -serial -serial_esc
! module /fiasco/sigma0
! module /genode/core
! module /genode/init
! module /config
! module /genode/timer
! module /genode/pci_drv
! module /genode/test-dde_linux26_net
The first four lines are L4/Fiasco specific. When using L4ka::Pistachio,
the 'menu.lst' entry looks like this:
! title Genode: DDE Linux 2.6 NET on L4/Pistachio
! kernel /pistachio/kickstart
! module /pistachio/x86-kernel
! module /pistachio/sigma0
! module /genode/core
! module /genode/init
! module /config
! module /genode/timer
! module /genode/pci_drv
! module /genode/test-dde_linux26_net
The web-server test requires the PCI bus driver and the timer service.
Therefore, the 'config' file for Genode's init should have following
content:
! <config>
! <start>
! <filename>timer</filename>
! <ram_quota>512K</ram_quota>
! </start>
! <start>
! <filename>pci_drv</filename>
! <ram_quota>512K</ram_quota>
! </start>
! <start>
! <filename>test-dde_linux26_net</filename>
! <ram_quota>16M</ram_quota>
! </start>
! </config>
Now, its time to create an ISO image from all files specified in
the GRUB configuration. For this, the new utility 'tool/create_iso'
becomes handy. The ISO image can then be booted on Qemu using the
following arguments:
! qemu -m 64 -serial stdio -no-kqemu -cdrom <iso-image> \
! -net nic,model=pcnet -net user -redir tcp:5555::80
The '-redir' argument tells qemu to redirect TCP connections with
localhost:5555 to the guest OS at port 80. After having booted
up Genode on Qemu, you can use your host's web browser to access
the web server:
! firefox http://localhost:5555
:Notes about using the TAP version:
* Preparations
* You must be permitted to sudo and have installed the tunctl
utility. Under Debian/Ubuntu execute
! sudo apt-get install uml-utilities
* Create TAP device
! TAPDEV=$(sudo tunctl -b -u $USER)
! sudo /sbin/ifconfig $TAPDEV 10.0.0.1
* setup DHCP server on $TAPDEV and 10.0.0.0/8
* Run qemu
! qemu -m 64 -serial stdio -no-kqemu -cdrom dde.iso \
! -net nic,model=pcnet \
! -net tap,ifname=$TAPDEV,script=no,downscript=no
* Ping
* Cleanup
* Stop DHCP server
* Remove TAP device
! sudo tunctl -d $TAPDEV
Operating-system services and libraries
#######################################
C Runtime
=========
We have replaced the 'malloc' implementation of the original FreeBSD C
library with a custom implementation, which relies on Genode's 'Heap' as
allocator. The FreeBSD libc reserves a default memory pool of 1MB, which
is no problem on FreeBSD because virtual memory is backed lazily with
physical pages on demand. On Genode however, we immediately account the
allocated memory, which implicates high quota requirements even for
applications that use little memory. In contrast, Genode's heap allocates
and accounts its backing store in relatively small chunks of a few KB.
Therefore, the quota accounting for applications is much more in line with
the actual memory usage. Furthermore, our custom 'malloc' implementation
has the additional benefit of being thread safe.
* Added i386-specific parts of gen lib, in particular longjmp, setjmp.
Device-Driver-Environment Kit
=============================
* The DDE Kit uses our alarm framework (located in the 'os' repository) now
rather than its own event-scheduler implementation formerly called 'Tick'.
* We refined the DDE Kit API and reduced the number of custom types. For
example, we removed the custom 'dde_kit_lock_t' and using
'struct dde_kit_lock' instead, and replaced 'dde_kit_thread_t' with
'struct dde_kit_thread'.
Because of the apparent stabilization of the DDE Kit API, we have now added
this API to Genode's official API reference.
[http://genode.org/documentation/api/dde_kit_index - See the documentation of the DDE Kit API...]
PS/2 input driver
=================
We improved the PS/2 keyboard driver by adding missing scan-code translations
for the scan code set 1, in particular the cursor keys.
Applications
############
Launchpad configuration
=======================
Launchpad is a graphical application for interactively starting and killing
programs. It is used for the default demonstration of Genode. By default,
launchpad displays a preconfigured list of programs and their respective
default memory quotas. The user can tweak the memory quota for each entry
with mouse and then start a program by clicking on its name. As an
alternative to using the default list, you can now define the list manually
by supplying a configuration to Launchpad. The following example tells
launchpad to display a list of two launcher entries:
!<config>
! <launcher>
! <filename>sdl_pathfind</filename>
! <ram_quota>10M</ram_quota>
! </launcher>
! <launcher>
! <filename>liquid_fb</filename>
! <ram_quota>10M</ram_quota>
! </launcher>
!</config>
To use this configuration for a Launchpad started via init, you can simply
insert the launchpad configuration into the '<start>' node of the launchpad
entry in init's 'config' file.
Platform-specific changes
#########################
L4/Fiasco
=========
* Raise 'Blocking_canceled' exceptions on canceled IPC calls
32-bit Linux
============
* We continued dissolving the dependency of Genode from the glibc by using
a custom 'getenv' implementation used during process creation.
* By default, we compile now with '-nostdinc' and explicitly specify
'/usr/include' as include search directory only when needed. Previously,
a Genode application, which included a host include file by mistake, has
not raised any compilation error when compiled for the Linux version of
Genode. Now, all Genode platforms behave equally with regard to include
search directories.
* We enforce using the actual compiler's C++ support libraries rather than
the default libraries installed on the host.
Tools and build infrastructure
##############################
Official tool chain
===================
At the download section of our website, we used to provide a crosstool-based
tool chain as pre-compiled binaries. Since we got several requests about
how to build such a tool chain from scratch, we created custom utility for
downloading, building, and installing the official Genode tool chain. You
can find the utility at 'tool/tool_chain'. For usage instructions, just
start 'tool_chain' without arguments. Because this utility is a plain script,
you can follow and verify each step that we use for creating the Genode tool
chain. Currently, this official tool chain is based on binutils 2.18 and
gcc 4.2.4.
As an alternative to installing the tool chain from source, we also
provide pre-compiled binaries at the download section of our website.
[http://genode.org/download/tool-chain - Visit our tool-chain download website...]
For the Linux version of Genode, we still use the host's default gcc
as tool chain. This way, we spare the hassle of downloading and installing
a custom tool chain for somebody who wants to give Genode a quick try.
With this is mind, we have fixes several small issues with gcc 4.3.2:
* Fixed dependency generation for gcc-4.3.2. Older version of gcc used to
append a '.o' dependency at the target of '.d'-files. However, gcc-4.3.2
seems to handle the option '-MT' differently, resulting in a rule that
contains only the '.d' as target. Now, we explicitly specify both the
'.o' file and the '.d' file as target. Consequently, on older gcc
versions, the '.o' file appears twice but that is no problem.
* Fixed assembler issue with the 'fnstsw' instruction in the C library.
This instruction does not accept eax but only ax as argument.
Build-directory creation tool
=============================
We added a rule for creating a pre-configured build directory for the
Pistachio version to our build-directory creation tool
('tool/builddir/create_builddir'). Furthermore, we changed the default
build configuration such that the official Genode tool chain is used for
L4/Fiasco and L4ka::Pistachio.
Build system
============
* Improved clean rule - visit each target directory only once
* Stop the build process on the first error by default, for continuing
the build process depite of an error, you can use the '-i' argument
of make.
* Compiler flags can now be set specific for compiling C and C++ sources.
This is needed because both variants allow different sets of warning
options. The new variables are called 'CC_CXX_OPT' and 'CC_C_OPT'.
ISO image creation tool
=======================
We have created a convenient front end for 'genisoimage', which we
use for testing Genode on Qemu. You can find this ISO-image-creation
tool at 'tool/create_iso'. For usage instructions, simply start the
tool without arguments.

585
doc/release_notes-09-05.txt
View File

@@ -1,585 +0,0 @@
==============================================
Release notes for the Genode OS Framework 9.05
==============================================
Genode Labs
Shortly after including support for the L4ka::Pistachio kernel in the
previous release, the Genode version 9.05 gives a further evidence of
Genode's high portability by making the framework available on top of
the OKL4 kernel. This kernel is a commercial-grade microkernel
developed by [http://ok-labs.com - Open Kernel Labs]. In Section
[Supporting the OKL4 kernel as new base platform], we elaborate on the
new base platform and summarize the experiences made during our porting
work.
The previous Genode release was accompanied by a source-code archive containing
the initial version of Qt4 for Genode. Our approach is to make the Qt4
framework available for building Genode applications running natively on the
microkernel rather than within a virtualization environment. As advertised in
our [http://genode.org/about/road-map - road map], we have now seamlessly
integrated the Qt4 framework into our mainline source tree. Furthermore, we
have adapted our port to the Qt4 version 4.5.1. Section [Integration of Qt4
into the mainline repository] gives a rough overview of the changes and an
introduction on how to use the Qt4 framework with Genode.
The third major feature of the release is the addition of preliminary USB
support. We have been able to port major parts of Linux' USB infrastructure
to Genode using the DDE Kit introduced in autumn 2008. Section [USB support]
presents an overview about the pursued design and the current state of
implementation.
Section [OKLinux on Genode] outlines our ongoing efforts of running Linux
as a node in Genode's process tree.
Supporting the OKL4 kernel as new base platform
###############################################
The OKL4 kernel developed by Open Kernel Labs started as a fork of the
L4ka::Pistachio kernel. Whereas L4ka::Pistachio remained true to the L4
x.2 specification, OKL4 was subject of major API changes geared towards high
performance on the ARM architecture. OKL4 earned much fame for executing a
user-level variant of Linux (OKLinux) on top the microkernel, which turned out
to be faster than executing Linux natively on the ARM9 architecture. Even
though OKL4 is primary targeted at the ARM architecture, we wanted to go for
the x86 variant because of two reasons. First, there exists the just mentioned
user-level port of Linux for OKL4, which looks like an attractive way to execute
Linux on Genode once Genode runs on OKL4. Second, we think that distributing
Genode in the form of ISO images bootable on plain PC hardware is the best
way to reach the OS community. Therefore, we decided to use OKL4 version 2.1 as
the base for our work. In contrast to later releases, this version supports
both x86 and ARM. The following section reviews the unique features of the
OKL4 kernel from our perspective.
OKL4 viewed from the angle of a Genode developer
================================================
On the kernel-API level, OKL4 has several interesting properties that had been
both welcome and challenging. We want to highlight the following points:
In contrast to prior L4 kernels, OKL4 has *removed wall-clock timeouts* from
the kernel interface. On L4, timeouts were used as arguments for for blocking
IPC operations serving two purposes. First, specifying IPC timeouts allowed the
IPC caller for regaining control over the blocking thread after the specified
time elapsed. This is considered as important in the case that the called
thread misbehaves and never answers the call. However, the problem of choosing
an appropriate timeout was never properly resolved. When dimensioning the
timeout too small, the called thread may miss the timeout just because it had
no chance to be selected by the scheduler in time. Such timeouts rely on the
presumption that there is low load on the system. On the other hand, when
dimensioning the timeout too high, the system will become sluggish when the
called thread misbehaves. For example, a simple GUI server may want to send
input events to its clients with a timeout to be robust against misbehaving
clients that never wait for events. When choosing a timeout too small, chances
are high that an event will occur at a time when the receiver is handling a
previous event. The timeout would trigger and the event would get lost. When
choosing the timeout too large, say 1 second, any misbehaving client could make
the GUI server freeze for 1 second. Therefore, timeouts for regaining control
over a blocked thread seem to be a bad idea. So we welcome their absence in
OKL4. The second use of timeouts is their use as user-level time source. On L4,
sleep is typically implemented as a blocking IPC with a timeout set to the
sleep value. For this purpose, a system built on top of OKL4 has to employ a
user level device driver accessing a timer device. In Genode, we already have a
timer service for this purpose. So we won't miss timeouts at all.
Classical L4 kernels provide two variants of *synchronous IPC*. So called long
IPC could copy any amount of memory from the sending to the receiving address
space. This is complicated operation because either communication partner may
specify communication buffers that contain unmapped pages. Hence, page faults
may occur during long-IPC operations. On L4, page faults, in turn, are handled
by the user land. Not until a user-level pager thread resolves the page fault
by establishing a mapping at the faulting address, the kernel can proceed the
IPC operation. This sounds pretty complicated, and it is. The second IPC
variant is called short IPC. It constrains the transferable payload to CPU
registers. Hence, these IPC operations should only be used for messages with a
payload of a maximum of 2 machine words. Because short IPCs are never touching
user-level memory pages, no page faults can occur.
On OKL4, there is only one IPC operation, which copies payload from the
sender's user-level thread-control block (UTCB) to the receiver's UTCB. An
UTCB is an always-mapped memory region. Hence no page faults can occur during
IPC operations. On Genode, the UTCB size of 256 bytes may become a limitation
when RPC messages get large. For example, session requests may include large
session-argument strings specifying session-constructor arguments. Current
services rely only on a few arguments so the size limitation is not an
apparent problem. But that may change for future services. Furthermore, in
contrast to L4 x.2, OKL4 does not allow for transferring payload other than
plain data. In particular, OKL4 does not support the transfer of memory
mappings via IPC. Removing memory mappings from the IPC operation is a very
good idea. On Genode, only roottask (core) establishes mappings and shared
memory is implemented as a user-level protocol (data spaces). There is no need
to allow arbitrary processes to establish memory mapping via IPC.
The *boot procedure* of OKL4 largely differs from other L4 kernels. This is
attributed to Open Kernel Labs' focus on embedded systems, which mostly rely on
single-image boot loading. OKL4 employs a tool (elfweaver) for creating a
bootable image from a bunch of files and an XML configuration file. Among the
declarations about which processes to be loaded and which policies to enforce,
the configuration file contains platform parameters such as the amount of
physical memory of the machine. This static approach to configure a system is
certainly useful for embedded systems but PC hardware uses to vary a lot. In
this case, evaluating boot-time memory descriptors would be the preferred
solution.
OKL4 introduces kernel support for *user-level synchronization*. Prior L4
kernels facilitated user-level synchronization through a combination of
synchronous IPC operations with either priorities or delayed preemption.
OKL4's mutexes can make the life in the user land much easier. However, we have
not looked into OKL4 mutexes yet.
There does not exist a recursive *map operation* as the source operand of the
map operation is a physical memory descriptor rather than a virtual address in
the mapper's address space. Consequently, this design eliminates the need for
having a recursive unmap operation and thereby, the need to maintain a mapping
data base in the kernel. This is cool because Genode keeps track of the
mappings at the user level anyway (within core). From our perspective, there is
no need to maintain mapping relationships in the kernel. Removing the mapping
database effectively discards a lot of much-discussed problems about how to
manage the mapping database in a clever way.
There exists *no root memory manager* (sigma0). Because the map operation
takes a physical memory descriptor as argument instead of a virtual address
in the mapper's address space. The mapper does not need to have the mapped
page locally mapped within its own address space. In fact, core (as the only
mapper in a Genode system) does only have very little memory mapped locally.
This design accelerates the boot time because there is no need to map each
physical page in core at startup as performed when running core on the other
L4 kernels.
These differences of OKL4 compared with the microkernels already supported
by Genode posed a number of interesting challenges and opportunities. We have
thoroughly documented the process in
[http://genode.org/documentation/articles/genode-on-okl4 - Bringing Genode to OKL4].
Usage
=====
For using Genode with OKL4, please refer to the following dedicated page:
:[http://genode.org/documentation/platforms/okl4 - Genode on the OKL4 microkernel]:
Site about building and using Genode with the OKL4 kernel.
Limitations of the current implementation
=========================================
The current implementation is able to execute the complete Genode demonstration
scenario on OKL4. This means, we can build and destroy arbitrary trees of
processes, have all the needed infrastructure in place to execute user-level
device drivers such as VESA and PS/2, perform inter-process communication
via RPC and shared memory, and have all basic framework API functions available.
We regard the current state as the first functional version. However, there are
the following points that need improvement and are subject to our future work.
:Incomplete timer driver:
On x86, the timer driver should program the PIT to dispatch sleep requests.
However, the I/O ports of the PIT can only by made available to one party in
the system (which naturally would be the timer driver). Unfortunately, there
are some VESA BIOSes around, which try using the PIT directly. The current
version of our VESA driver does not virtualize these accesses. It rather
tries to gain direct access to the I/O ports from core. This would not work
if the timer already uses this device resource. Our plan is to supplement
our VESA driver with a virtual PIT that uses the timer service as back end.
Then we can safely use the PIT by the timer driver.
:Signalling framework not yet implemented:
We have not yet implemented Genode's API for asynchronous notifications
in the OKL4 version. In fact, the goal of the initial version of the
OKL4 support was running the default demonstration scenario, which does
not rely on signals. The second and more technical reason is that we
consider exploiting OKL4's event mechanism for implementing the signalling
API but have not finalized the design. The generic implementation as used
on the other platforms cannot be used on OKL4 because this implementation
utilizes one helper thread per signal transmitter. Within core, each RM
session is a potential signal transmitter, which means that we need to
create a helper thread per process. Unfortunately, by default, OKL4
limits the number of threads within roottask (core) to only 8 threads,
which would impose a severe limit on the number of processes we could
run on OKL4.
:OKL4's kernel mutexes yet to be used:
We have not yet explored the use of mutexes provided by the OKL4 kernel
for implementing Genode synchronization APIs but we rather rely on a
yielding spin lock for now. This has a number of drawbacks such as high
wake-up latencies in the contention case (depending on the scheduling
time slice), no support for priorities, and no fairness. Although it
is a simple and robust solution to start with, we plan to facilitate
the OKL4 kernel feature with our upcoming work.
:Overly simplistic UTCB allocation:
Right now, we allocate a fixed amount of 32 UTCBs per address space and
thereby limit the maximum number of threads per process. In the future,
this limit should be made configurable.
:Managed dataspaces not yet supported:
The support of managed dataspaces relies on the signal API, which is
not yet available for OKL4.
:Message buffers are limited to 256 bytes:
Because OKL4 performs message-based inter-process communication by
copying data between the UTCBs of the communicating threads, the
UTCB size constaints the maximum message size. Therefore, message
must not exceed 256 bytes. This is not a huge problem for the currently
available Genode programs but we can imagine session argument-lists
to become larger in the future.
:Advanced thread functions are incomplete:
Thread functions such as querying registers of remote threads are not yet
implemented.
Integration of Qt4 into the mainline repository
###############################################
Qt4 is a tool kit for developing platform-independent applications. It
comprises a complete platform-abstraction layer and a rich GUI tool kit
widely used for commercial and open-source applications. It is particularly
known as the technical foundation of the KDE project. The previous Genode
release was accompanied by a snapshot of our initial port of Qt4 to Genode. For
the current release, we have turned this proof-of-concept implementation into a
properly integrated part of the Genode mainline development. This enables Qt4
applications to be executed natively on the full range of kernels supported by
Genode.
Usage
=====
We complemented Genode's source tree with the new 'qt4' source-code repository,
which contains the Genode-specific parts of the Qt4 framework. The most
portions for the Qt4 framework are used unmodified and thereby have not been
made part of the Genode source tree. Instead, we fetch the original Qt4 source
code from Trolltech's FTP server. This way, our source tree remains tidy and
neat.
For using Qt4 for your Genode applications, you first need to download and
prepare the original Qt4 source codes and build a few Qt4 tools such as the
meta-object compiler and the resource compiler. The makefile found in the
top-level directory of the 'qt4' repository automates this task:
! make prepare
To include the 'qt4' repository into the Genode build process, just add the
'qt4' directory to the 'REPOSITORIES' declaration of the 'etc/build.conf' file
within your build directory. Make sure that the repositories 'demo' and 'libc'
are included as well. The 'qt4' repository comes with a couple of demo applications.
The 'qt_launchpad' is especially interesting because it makes use of both the
Qt4 framework and the Genode framework in one application.
Features and limitations
========================
The Qt4 port comprises Qt's Core library, GUI library, Script library, XML
library, and the UI tools library.
For using Qt4 on the Linux version of Genode, we recommend using the Genode
tool chain rather than your host's tool chain. Qt4 makes use of a lot of libc
functionality, supplied by Genode's 'libc' repository. However, on Linux we
still link against your host's libc. This becomes a problem if your host
compiler's C++ support code references libc functionality that is not part of
Genode's libc. Thereby the linker will silently create references to glibc
symbols, making both libraries collide. So if using Qt4, we recommend using the
Genode tool chain:
:[http://genode.org/download/tool-chain]:
Information about downloading and using the Genode tool chain
USB support
###########
This release introduces the first fragments of USB support to Genode, taking
the USB human-interface device (HID) class as starting point. With this work,
we follow our approach of reusing unmodified Linux device drivers executed
within a device-driver environment called DDE Linux. In the previous release,
we already utilized this approach for realizing basic networking on Genode.
With this release, we complement DDE Linux with support required by USB
drivers. We are grateful for being able to base our implementation on the
excellent foundation laid by Dirk Vogt. He described his work in
[http://os.inf.tu-dresden.de/papers_ps/beleg-vogt.pdf - USB for the L4 environment].
For USB HID support, we added the Linux USB and input subsystems to the DDE
Linux 2.6 framework. Besides the 'dde_linux26/net.h' API for network drivers
added in Genode 9.02, the current version also includes APIs for input
('dde_linux26/input.h') and USB ('dde_linux26/usb.h'). We intend these
interfaces to mature towards generic driver-library APIs in the future. For
example, BSD-based drivers shall transparently provide the same functionality
as the current Linux drivers, which permits the simple reuse of driver server
implementations.
[image usb_current]
Image [usb_current] illustrates the current implementation of the USB-based
human-interface device (HID) driver. In this monolithic setup, all parts of the
USB stack and the device API are executed within one address space. These parts
are
* Input server glue code
* HID driver and input subsystem
* Core functions for management of USB request buffers (URBs),
attached devices, and registered drivers
* Host controller drivers for UHCI, OHCI, and EHCI
[image usb_aspired]
We regard this as an intermediate step towards our goal to decompose the USB
stack. Image [usb_aspired] shows our aspired design. In this design, the
USB server and one or more USB gadget drivers run in dedicated address spaces.
The USB server provides two interfaces called USB session interface and USB
device interface. A USB session interface corresponds to a virtual root hub,
from which USB devices can be queried. The client of the USB session interface
is usually an USB gadget driver that uses the USB device interface. Because
this interface is used for transferring the actual payload at a potentially
high bandwidth, it is based on shared memory and signals. The USB server
consists of the following components:
* USB server glue code
* Virtual USB device driver managing all attached devices
* Core functions including hardware hub management
* Host controller drivers
The USB server presents a virtual USB hub to each USB gadget driver. Such
a driver consists of:
* Device interface, e.g., input server glue code
* Gadget driver, e.g., HID driver and input subsystem
* Core functions
* Virtual host controller
* USB client glue code
The HID driver uses the USB session API to monitor ports of its virtual root
hub and submit URBs to attached devices. The session interface facilitates the
signalling framework for event notification and a shared-memory dataspace for
URB transmission.
The 'os' repository already contains the USB session and USB device interfaces.
However, the decomposition is not yet in a functional state.
:Current limitations:
The current monolithic implementation of the USB HID service can already be
used as a replacement of the PS/2 driver. However, both drivers cannot be used
at the same time, yet. To enable the use of both USB HID and PS/2, we plan to
create a further component that merges multiple streams of input events and
passes the result to the GUI server.
OKLinux on Genode
#################
According to our road map, we pursued the goal to run Linux as a node in
Genode's process tree. We explored two approaches:
:Reanimating the Afterburner project conducted by the [http://l4ka.org - L4Ka group]:
This approach is the result of the L4Ka groups's long-year experience with
manually supporting L4Linux on top of the L4ka::Pistachio kernel. Because of
the high costs of maintaining the paravirtualized Linux kernel, a
semiautomatic paravirtualization technique was created. According to the
impressive results presented in
[http://www.l4ka.org/l4ka/publ_2006_levasseur-ua_soft-layering.pdf - Pre-Virtualization: Soft Layering for Virtual Machines],
this approach is able to drastically reduce maintenance costs while retaining
good performance. Furthermore, the approach was applied not only to Linux
running on the L4 kernel but also for using Xen or Linux as underlying
host operating systems.
:Porting the OKL4-specific version of L4Linux to Genode:
Open Kernel Labs maintain a custom version of L4Linux that runs on OKL4. This
version is mostly referred to as OKLinux aka Wombat. Since Genode can now use OKL4
as base platform, the reuse of OKLinux in combination with Genode has become
a feasible option.
Both approaches have pros and cons. Whereas Afterburner is a intriguing
approach, this project seems to be stalled. It relies on a rather old tool
chain, and recent Linux features such as thread-local storage support are not
considered, yet. To pick up this solution for Genode will require us to fully
understand the mechanisms and the code. So we consider this as a mid-term
solution. In short term, running OKLinux on Genode is more feasible. We were
already able to create a prototype version of OKLinux running on Genode. This
version starts up the kernel including all Linux kernel threads, mounts the
boot partition supplied as memory image, and starts the init process. The
engineering costs had been rather low. We replaced the Iguana user land
libraries as originally used by Wombat by a Genode-specific reimplementation to
keep our manual adaptions of the Linux kernel code as small as possible.
Our custom reimplementation of the needed Iguana APIs consists of less than
1,000 lines of code (SLOC). The diff for our changes to the OKLinux kernel code
comprises less than 1,000 lines. We plan to make a snapshot of this prototype
publicly available soon.
Operating-system services and libraries
#######################################
Nitpicker GUI server
====================
We optimized the performance of the 'refresh' call, which updates all views of
a session, which display a given buffer portion. The new implementation restricts
the redraw operations to the fragment of each view that displays the specified
buffer portion. The performance improvement becomes most visible when updating
only small parts of a buffer.
USB session interface
=====================
Genode's emerging USB support introduces two new interfaces to the 'os' repository,
which are called USB session and USB device.
An _USB_session_ is a virtual USB bus with just one root hub with 'MAX_PORTS'
downstream ports. The client of such as session submits USB request blocks
(URBs) and is, in turn, informed about port changes on the root hub as well as
request completion. Connected USB devices can be referenced by USB device
capabilities and are associated with one port at the virtual root hub on
server side.
An _USB_device_ references a hardware device connected to a virtual USB bus's
root hub. The device capability enables the client to send USB request
blocks to the hardware device.
Input interface
===============
We updated the key codes of the input interface in response to recent changes
of Linux' 'dev/event' definitions.
VESA driver
===========
Until now, there existed different processes that tried to access the PCI bus
via I/O ports, in particular the VESA framebuffer driver and the PCI bus
driver.
Since core enforces that each I/O port can only be assigned exclusively to one
component in the system, multiple processes that need access to the same I/O
ports cannot run at the same time. For our default demonstration scenario, we
had been able to allow the VESA driver to use the PCI I/O ports because nobody
else needed them. However, our growing base of device drivers relies on the
PCI bus driver. To be able to use the VESA driver together with other drivers,
we virtualized the access to the PCI bus from within the VESA driver.
Our current PCI virtualization code is pretty limited. The VESA driver sees a
virtual PCI bus with only the VGA card attached. For now, we only allow reading
the PCI configuration space of this device, but not writing to it. Apparently,
this simple approach is sufficient to run the VESA BIOS of Qemu. However, other
VESA BIOS implementations may need further access to the PCI device's
configuration space. For example, for querying the size of a PCI resource,
write access to base address registers is required. In such an event, the VESA
driver will print a message about the missing virtualization feature:
! writing data register not supported
If you see such a message, we are very interested to see your log output such
that we can enhance our PCI virtualization code as needed. Please contact us!
Base framework
##############
In the process of bringing Genode to the OKL4 kernel, we have generalized much
of former platform-specific code:
* The initialization of C++ exception handling has now become part of the
generic 'cxx' library in the 'base' repository. All platforms except
Linux are using this generic library now.
* The 'server' library used to contain a platform-specific part that
implemented the 'manage' function of a 'Server_entrypoint'. The
generalized version of this library is now being used on all platforms
other than Linux.
* We unified core-internal interfaces and their implementations such as
'Dataspace_component', 'Cap_session_component', 'Rm_session_component',
and 'Irq_session_component'. The result has become part of the 'base'
repository.
* On OKL4, threads need to execute small startup code for querying their
own thread IDs. Therefore, we have extended the 'Thread_base' interface
with a platform-specific hook function called '_thread_bootstrap'.
* The types defined in 'base/native_types.h' had been complemented by a
new 'Native_thread_id' type. This type is exclusively used by core and the
framework libraries. For using the Genode API, this type is meaningless.
* For the 64bit support, we slightly refined the interfaces of some utility
template functions in 'util/misc_math.h'. Furthermore, parts of the generic
marshalling code of the IPC framework needed refinement, but no API changes
were needed.
Linux-specific changes
######################
Adaptation to 64 bit
====================
Because most Genode developers tend to work with the Linux version of Genode,
supporting 64-bit Linux becomes increasingly important. With the current release,
we start to officially support 64-bit Linux as base platform. This comes
along with the following changes:
* We replaced the 'spec-x86.mk' file with new 'spec-x86_32.mk' and 'spec-x86_64.mk'
files. The default version of 'base-linux/etc/specs.conf' automatically
chooses the right spec file according to the output of 'uname -m'. Therefore,
output of the build processes matches your host architecture. This behaviour
can be changed by placing a customized 'spec.conf' file in your build directory's
'etc/' subdirectory.
* We added type definitions for 64-bit-specific fixed-size integers in the form
of a 64-bit-specific 'fixed_stdint.h' file.
* Because using 64 bit instead of 32 bit changes the payload size of RPC
messages, we had to adjust several message buffers such as 'Ram_session_client'
and 'Input::Session_client', and adapted the used stack sizes.
* Towards the goal of completely dissolving Genode's dependency on the Linux' glibc,
we implemented custom system-call bindings. Apparently, Linux' syscall interface
differs between 32 bit and 64 bit. For example, the 32-bit version handles
all socket-related calls via a compound 'socketcall' whereas the 64-bit
version uses distinct syscalls. Another difference is the handling of the
'mmap' syscall and different behaviour of 'tkill'. The latter problem was
resolved by replacing 'tkill' with 'tgkill' and setting the thread-group
argument of the corresponding PID. Therefore, a 'Native_thread_id' on Linux
now comprises both the TID and the PID.
* The 'Platform_env' on Linux contains a local implementation of the 'Rm_session'
interface, which uses 'mmap' to attach dataspaces to the process' address
space and maintains the region list in the form of a static array. This array
was dimensioned to 256 entries, which constrained the maximum amount of
usable memory when allocating a lot of small blocks via Genode's heap. Since
the heap allocates backing store at the granularity of only 16KB, the worst
case for reaching this limit was about 4MB. This was OK for our simple test
applications. But for using Qt4, in particular on 64 bit, this has become a
serious limitation. For now, we increased the region limit to 4096 and plan
to replace the static array with a dynamically growing data structure.
Furthermore, we made the heap granularity depend on the actual machine-word
size. Therefore, the heap allocates its backing store in 32KB blocks when
running on 64 bit.
Debugging hooks
===============
On Linux, we use gdb for debugging Genode. This is feasible as long as the
targeted process is running. However, during low-level debugging, we had the
recurring problem of a thread dying shortly after starting up. So we added a hook
for halting a thread at the startup in order to be able to attach gdb to the
thread before it dies. This simple hook lets the thread wait for a key press by
directly calling the 'read' syscall. We also added a simple debug facility for
printing debug messages bypassing Genode's LOG service by calling the 'write'
syscall directly. Both hooks are now part of the Linux version of the 'env'
library (see 'base-linux/src/base/env/debug.cc'). Note that these hooks are not
part of the Genode API. There exists no header file.

1016
doc/release_notes-09-11.txt
View File

File diff suppressed because it is too large Load Diff

1224
doc/release_notes-10-02.txt
View File

File diff suppressed because it is too large Load Diff

1211
doc/release_notes-10-05.txt
View File

File diff suppressed because it is too large Load Diff

871
doc/release_notes-10-11.txt
View File

@@ -1,871 +0,0 @@
===============================================
Release notes for the Genode OS Framework 10.11
===============================================
Genode Labs
During the past three months, the Genode project was primarily driven by our
desire to create a bigger picture out of the rich set of components that we
introduced over time, in particular over the last year. Looking back at the
progress made since mid 2009, there were many functional additions to the
framework, waiting to get combined. To name a few, we added support for
networking, audio output, real-time priorities, mandatory access control,
USB, ATAPI block devices, Python, hardware-accelerated 3D graphics, Qt4,
the WebKit-based Arora browser, and the paravirtualized OKLinux kernel.
So many wonderful toys waiting to get played with. This is how the idea of
creating [http://genode.org/download/live-cds - the new Genode Live CD] was
born. In the past, Genode was mostly used in settings with a relatively static
configuration consisting of several components orchestrated to fulfill
a few special-purpose functions. Now, the time has come for the next step,
creating one dynamic setup that allows for the selection of different subsystems
at runtime rather than at boot time.
This step is challenging in several ways. First, the processes that form
the base system have to run during the entire time of all demo setups. If
any of those processes contained stability problems or leaked memory, it would
subvert the complete system. Second, the components of all subsystems combined
are far too complex to be loaded into memory at boot time. This would not
only take too long but would consume a lot of RAM. Instead, those components
and their data had to be fetched from disk (CDROM) on demand. Third, because
multiple demo subsystems can be active at a time, low-level resources such as
networking and audio output must be multiplexed to prevent different
subsystems from interfering with each other. Finally, we had to create a
single boot and configuration concept that is able to align the needs of all
demos, yet staying manageable.
Alongside these challenges, we came up with a lot of ideas about how Genode's
components could be composed in new creative ways. Some of these ideas such
as the browser-plugin concept and the http-based block server made it onto
the Live CD. So for producing the Live CD, we not only faced the said
technical challenges but also invested substantial development effort in new
components, which contributed to our overall goal. Two weeks ago, we released
the Live CD. This release-notes document is the story about how we got there.
To keep ourself focused on the mission described above, we deferred the
original roadmap goal for this release, which was the creation of a Unix-like
runtime environment to enable compiling Genode on Genode. This will be the
primary goal for the next release.
Execution environment for gPXE drivers
######################################
Up to now, DDE Linux provided Genode with drivers for hardware devices
ranging from USB HID to WLAN. In preparation of the live CD, we
noticed the demand for support of a broader selection of ethernet
devices. Intel's e1000 PCI and PCIe cards seemed to mark the bottom
line of what we had to support. The major advantage of NIC drivers
from Linux is their optimization for maximum performance. This emerges
a major downside if DDE Linux comes into play: We have to provide all
the nifty interfaces used by the driver in our emulation framework. To
achieve our short-term goal of a great live CD experience, we had to
walk a different path.
[http://gpxe.org/ - gPXE] is a lovely network boot loader / open-source
PXE ROM project and the successor of the famous Etherboot
implementation. Besides support for DNS, HTTP, iSCSI and AoE, gPXE
includes dozens of NIC drivers and applies a plain driver framework.
As we were also itching to evaluate DDE kit and the DDE approach at
large with this special _donator OS_, we went for implementing the
device-driver environment for gPXE (DDE gPXE).
The current version provides drivers for e1000, e1000e, and pcnet
devices. The emulation framework comprises just about 600 lines of
code compared to more than 22,000 LOC reused unmodified from gPXE.
Benchmarks with the PCNet32 driver showed that DDE gPXE's performance
is comparable to DDE Linux.
The gPXE driver environment comes in the form of the new 'dde_gpxe'
repository. For building DDE gPXE, you first need to download and patch
the original sources. The top-level makefile of this repository automates
this task. Just issue:
! make prepare
Now, you need to include the DDE gPXE repository into your Genode
build process. Just add the path to this directory to the
'REPOSITORIES' declaration of the 'etc/build.conf' file within your
build directory, for example
! REPOSITORIES += $(GENODE_DIR)/dde_gpxe
After successful build the DDE gPXE based ethernet driver is located
at 'bin/gpxe_nic_drv'.
On-demand paging
################
In the [http://genode.org/documentation/release-notes/8.11#section-8 - release 8.11],
we laid the foundation for implementing user-level dataspace managers.
But so far, the facility remained largely unused except for managing thread
contexts. This changed with this release.
So what is a user-level dataspace manager and who needs it? In short,
Genode's memory management is based on dataspaces. A dataspace is a
container for memory. Normally, it is created via core's RAM or ROM
services. The RAM service hands out dataspaces containing contiguous
physical memory. After allocating such a RAM dataspace, the creator can
attach the dataspace to its own address space to access the dataspace
content. In addition, it can pass a dataspace reference (called dataspace
capability) to other processes, which, in turn, can than attach the same
dataspace to their local address space, thereby establishing shared memory.
Similarly, core's ROM service hands out boot-time binary data as dataspaces.
For the most use cases of Genode so far, these two core services were the
only dataspace providers needed. However, there are use cases that require
more sophisticated memory management. For example, to implement swapping,
the content of a dataspace must be transferred to disk in a way that
is transparent to the users of the dataspace. In monolithic kernels, such
functionality is implemented in the kernel. But on a multi-server OS
such as Genode, this is no option. Implementing such a feature into
core would increase the trusted computing base of all applications
including those who do not need swapping. Core would need a hard-disk
driver, effectively subverting the Genode concept. Other examples for
advanced memory-management facilities are copy-on-write memory and
non-contiguous memory - complexity we wish to avoid at the root of the
process tree. Instead of implementing such memory management facilities
by itself, core provides a mechanism to let any process manage dataspaces.
This technique is also called user-level page-fault handling.
For the Live CD, we decided to give Genode's user-level page-fault handling
facility a go. The incentive was accessing files stored on CDROM in an
elegant way. We wanted to make the CDROM access completely transparent to
the applications. An application should be able to use a ROM session as
if the file was stored at core's ROM service. But instead of being
provided by core, the session request would be delegated to an
alternative ROM service implementation that reads the data from disk
as needed. Some of the files stored in the CDROM are large. For example,
the disk image that we use for the Linux demo is 160MB. So reading
this file at once and keeping it in memory is not an option. Instead, only
those parts of the file should be read from disk, which are actually
needed. To uphold the illusion of dealing with plain ROM files for
the client, we need to employ on-demand-paging in the CDROM server.
Here is how it works.
# The dataspace manager creates an empty managed dataspace. Core
already provides a tool for managing address spaces called
region manager (RM service). A RM session is an address space,
to which dataspaces can be attached. This is exactly what is
needed for a managed dataspace. So a dataspace manager uses the
same core service to define the layout of a managed dataspace
as is used to manage the address space of a process. In fact,
any RM session can be converted into a managed dataspace.
! enum { MANAGED_DS_SIZE = 64*1024*1024 };
! Rm_connection rm(0, MANAGED_DS_SIZE);
This code creates a RM session with the size of 64MB. This is an empty
address space. A dataspace capability that corresponds to this address
space can then be requested via
! Dataspace_capability ds = rm.dataspace();
# The dataspace capability can be passed to a client, which may
attach the dataspace to its local address space. Because the
managed dataspace is not populated by any backing store, however,
an access would trigger a page fault, halting the execution of
the client. Here, the page-fault protocol comes into play.
# The dataspace manager registers itself for receiving a signal each time
a fault occurs:
! Signal_receiver rec;
! Signal_context client;
! Signal_context_capability sig_cap = rec.manage(client);
! rm.fault_handler(sig_cap);
When an empty part of the managed dataspace is accessed by any
process, a signal is delivered. The dataspace manager can then
retrieve the fault information (access type, fault address) and
dispatch the page fault by attaching a real dataspace at the
fault address of the managed dataspace. In a simple case, the code
looks as follows:
! while (true) {
! Signal signal = rec.wait_for_signal();
! for (int i = 0; i < signal.num(); i++) {
! Rm_session::State state = rm.state();
! ds = alloc_backing_store_dataspace(PAGE_SIZE);
! rm.attach_at(ds, state.addr & PAGE_MASK);
! }
! }
This simple page-fault handler would lazily allocate a page of
backing store memory each time a fault occurs. When the backing
store is attached to the managed dataspace, core will automatically
wake up the faulted client.
# The example above has the problem that the dataspace manager has
to pay for the backing store that is indirectly used by the client.
To prevent the client from exhausting the dataspace manager's memory,
the dataspace manager may choose to use a limited pool of backing
store only. If this pool is exceeded, the dataspace manager can reuse
an already used backing-store block by first revoking it from its
current managed dataspace:
! rm.detach(addr);
This will flush all mappings referring to the specified address
from all users of the managed dataspace. The next time, this
address region is accessed, a new signal will be delivered.
This page-fault protocol has the following unique properties. First,
because core is used as a broker between client and dataspace manager, the
dataspace manager remains completely unaware of the identity of its client.
It does not even need to possess the communication right to the client. In
contrast, all other user-level page-fault protocols that we are aware of
require direct communication between client and dataspace manager. Second,
because dataspaces are used as first-level objects to resolve page faults,
page faults can be handed at an arbitrary granularity (of course, a multiple
of the physical page size). For example, a dataspace manager may decide to
attach backing-store dataspaces of 64K to the managed dataspace. So the
overhead produced by user-level page-fault handler can be traded for the
page-fault granularity. But most importantly, the API is the same across
all kernels that support user-level page fault handling. Thus the low-level
page-fault handling code becomes inherently portable.
Having said that, we have completed the implementation of the described
core mechanisms, in particular the 'detach' facility, for OKL4. The ISO9660
driver as featured on the Live CD implements the 'ROM' interface and
reads the contents of those files from CDROM on demand. It uses a
fixed pool of backing store, operates at a page-fault granularity of
64KB, and implements a simple fifo replacement strategy.
Base framework
##############
There had been only a few changes to the base framework described as
follows.
We unified the core-specific console implementation among all
base platforms and added synchronization of 'vprintf' calls.
The kernel-specific code resides now in the respective
'base-<platform>/src/base/console/core_console.h' files.
We removed the argument-less constructor from 'Allocator_avl_tpl'.
This constructor created an allocator that uses itself for
meta-data allocation, which is the usual case when creating
local memory allocators. However, on Genode, this code is typically
used to build non-memory allocators such as address-space regions.
For these use cases, the default policy is dangerous. Hence, we
decided to remove the default policy.
The 'printf' helper macros have been unified and simplified. The
available macros are 'PINF' for status information, 'PWRN' for warnings,
'PLOG' for log messages, and 'PERR' for errors. By default, the message
types are colored differently to make them easily distinguishable.
In addition to normal messages, there is the 'PDBG' for debugging
purposes. It remains to be the only macro that prints the function name
as message prefix and is meant for temporary messages, to be removed
before finalizing the code.
Genode's on-demand-paging mechanism relies on the signalling framework.
Each managed dataspace is assigned to a distinct signal context.
Hence, signal contexts need to be created and disposed alongside
with managed dataspaces. We complemented the signalling framework
with a 'dissolve' function to enable the destruction of signal
contexts.
Operating-system services and libraries
#######################################
Finished transition to new init concept
=======================================
With the release 10.05, we introduced the
[http://genode.org/documentation/release-notes/10.05#section-0 - current configuration concept of init].
This concept supports mandatory access control and provides flexible
ways for defining client-server relationships. Until now, we maintained
the old init concept. With the current release, the transition to the
new concept is finished and we removed the traditional init.
We retained the support for loading configurations for individual subsystems
from different files but adopted the syntax to the use of attributes.
Instead of
! <configfile>subsystem.config</configfile>
the new syntax is
! <configfile name="subsystem.config"/>
Virtual network bridge (Proxy ARP)
==================================
Since we originally added networking support to Genode, only one program could
use the networking facilities at a time. In the simplest form, such a program
included the network driver, protocol stack, and the actual application. For
example, the uIP stack featured with release 9.02 followed this approach.
In release 9.11 we added the 'Nic_session' interface to decouple the network
driver from the TCP/IP protocol stack. But the 1-to-1 relation between
application and network interface remained. With the current release, we
introduce the 'nic_bridge' server, which is able to multiplex the 'Nic_session'
interface.
The implementation is roughly based on the proxy ARP RFC 1027. At startup, the
'nic_bridge' creates a 'Nic_session' to the real network driver and, in turn,
announces a 'Nic' service at its parent. But in contrast to a network driver
implementing this interface, 'nic_bridge' supports an arbitrary number of
'Nic_sessions' to be opened. From the client's perspective, such a session
looks like a real network adaptor.
This way, it has become possible to run multiple TCP/IP stacks in
parallel, each obtaining a distinct IP address via DHCP. For example,
is has become possible to run multiple paravirtualized Linux kernels
alongside an lwIP-based web browser, each accessing the network via a
distinct IP address.
As a side effect for developing the 'nic_bridge', we created a set
of utilities for implementing network protocols. The utilities are
located at 'os/include/net' and comprise protocol definitions for
ethernet, IPv4, UDP, ARP, and DHCP.
Nitpicker GUI server
====================
Our work on the Live CD motivated several improvements of the Nitpicker
GUI server.
Alpha blending
~~~~~~~~~~~~~~
In addition to nitpicker's plain pixel buffer interface that is compatible
with a normal framebuffer session, each nitpicker session can now have
an optional alpha channel as well as an corresponding input-mask channel
associated. Both the alpha channel and the input mask are contained in the
same dataspace as the pixel buffer. The pixel buffer is followed by
the 8-bit alpha values, which are then followed by the input-mask values.
This way, the presence of an alpha channel does not interfere with the
actual pixel format. Each 8-bit input mask value specifies the user-input
policy for the respective pixel. If the value is zero, user input
referring to the pixel is not handled by the client but "falls through"
the view that is visible in the background of the pixel. This is typically
the case for drop shadows. If the input-mask value is '1', the input
is handled by the client.
With the input-mask mechanism in place, we no longer have a definitive
assignment of each pixel to a single client anymore. In principle, an
invisible client is able to track mouse movements by creating a full-screen
view with all alpha values set to '0' and all input-mask values set to '1'.
Once, the user clicks on this invisible view, the user input gets routed
to the invisible client instead of the actually visible view. This
security risk can be addressed at two levels:
* In X-Ray mode, nitpicker completely disables alpha blending
and the input-mask mechanism such that the user can identify the
client that is responsible for each pixel on screen.
* The use of the alpha channel is a session argument, which is specified
by nitpicker clients at session-creation time. Consequently, this
session argument is subjected to the policy of all processes involved
with routing the session request to nitpicker. Such a policy may permit
the use of an alpha channel only for trusted applications.
_Caution:_ The use of alpha channels implies read operations from
the frame buffer. On typical PC graphics hardware, such operations are
extremely slow. For this reason, the VESA driver should operate in
buffered mode when using alpha blending in Nitpicker.
Tinted views in X-Ray mode
~~~~~~~~~~~~~~~~~~~~~~~~~~
We added support for tinting individual clients or groups of clients
with different colors based on their label as reported at session-creation
time. By using session colors, nitpicker assists the user to tell apart
different security domains without reading textual information. In
addition to the tinting effect, the title bar presents the session
color of the currently focused session.
The following nitpicker configuration tints all views of the launchpad
subsystem in blue except for those views that belong to the testnit
child of launchpad. Those are tinted red.
! <config>
! <policy label="launchpad" color="#0000ff"/>
! <policy label="launchpad -> testnit" color="#ff0000"/>
! </config>
Misc Nitpicker changes
~~~~~~~~~~~~~~~~~~~~~~
We introduced a so-called 'stay-top' session argument, which declares
that views created via this session should stay on top of other views.
This function is useful for menus that should always remain accessible
or banner images as used for Live CD.
Nitpicker's reserved region at the top of the screen used to cover up
the screen area as seen by the clients. We have now excluded this area
from the coordinate system of the clients.
We implemented the 'kill' mode that can be activated by the 'kill' key.
(typically this is the 'Print Screen' key) This feature allows the user
to select a client to be removed from the GUI. The client is not
actually killed but only locked out. The 'kill' mode is meant as an
emergency brake if an application behaves in ways not wanted by the
user.
ISO9660 server
==============
As outlined in Section [On-demand paging], we revisited the ISO9660 server
to implement on-demand-paged dataspaces. It is the first real-world
use case for Genode's user-level page-fault protocol. The memory pool
to be used as backing store for managed dataspaces is dimensioned according
to the RAM assigned to the iso9660 server. The server divides this backing
store into blocks of 64KB and assigns those blocks to the managed dataspaces
in a fifo fashion. We found that using a granularity of 64KB improved the
performance over smaller block sizes because this way, we profit from reading
data ahead for each block request. This is particularly beneficial for
CDROM drives because of their extremely long seek times.
Audio mixer
===========
We added a new *channel synchronization* facility to the 'Audio_out_session'
interface. An 'Audio_out_session' refers to a single channel. For stereo
playback, two sessions must be created. At session-creation time, the
client can provide a hint about the channel type such as "front-left" as
session-construction argument. This design principally allows for supporting
setups with an arbitrary amount of channels. However, those channels must
be synchronized. For this reason, we introduced the 'sync_session' function
to the 'Audio_out_session' interface. It takes the session capability of
another 'Audio_out_session' as argument. The specified session is then
used as synchronization reference.
To reduce the latency when stopping audio replay, we introduced a new *flush*
function to the 'Audio_out_session' interface. By calling this function,
a client can express that it is willing to discard all audio data already
submitted to the mixer.
Furthermore, we improved the audio mixer to support both long-running
streams of audio and sporadic sounds. For the latter use case, low latency
is particularly critical. In this regard, the current implementation is a
vast improvement over the initial version. However, orchestrating the
mixer with audio drivers as well as with different clients (in particular
ALSA programs running on a paravirtualized Linux) is not trivial. In the
process, we learned a lot, which will eventually prompt us to further
optimize the current solution.
Nitpicker-based virtual Framebuffer
===================================
To support the browser-plugin demo, we introduced 'nit_fb', which is a
framebuffer service that uses the nitpicker GUI server as back end. It
is similar to the liquid framebuffer as featured in the 'demo' repository
but in contrast to liquid framebuffer, 'nit_fb' is non-interactive.
It has a fixed screen position and size. Furthermore, it does not
virtualize the framebuffer but passes through the framebuffer portion of
the nitpicker session, yielding better performance and lower latency.
If instantiated multiple times, 'nit_fb' can be used to statically arrange
multiple virtual frame buffers on one physical screen. The size and screen
position of each 'nit_fb' instance can be defined via Genode's configuration
mechanism using the following attributes of the 'nit_fb' config node:
! <config xpos="100" ypos="150"
! width="300" height="200"
! refresh_rate="25"/>
If 'refresh_rate' isn't set, the server will not trigger any refresh
operations by itself.
On the Live CD, each browser plugin instantiates a separate instance of
'nit_fb' to present the plugin's content on screen. In this case, the
view position is not fixed because the view is further virtualized by the
loader, which imposes its policy onto 'nit_fb' - Genode's nested
policies at work!
TAR ROM service
===============
For large setups, listing individual files as boot modules in single-image
creation tools (e.g., elfweaver) or multiboot boot loaders can be
cumbersome, especially when many data files or shared libraries are
involved. To facilitate the grouping of files, 'tar_rom' is an
implementation of the 'ROM' interface that operates on a 'tar' file.
The name of the TAR archive must be specified via the 'name' attribute of
an 'archive' tag, for example:
! <config>
! <archive name="archive.tar"/>
! </config>
The backing store for the dataspaces exported via ROM sessions is accounted
on the 'rom_tar' service (not on its clients) to make the use of 'rom_tar'
transparent to the regular users of core's ROM service. Hence, this service
must not be used by multiple clients that do not trust each other.
Typically, 'tar_rom' is instantiated per client.
The Live CD uses the 'tar_rom' service for the browser demo. Each plugin
is fetched from the web as a tar file containing the config file of the
plugin subsystem as well as supplemental binary files that are provided
to the plugin subsystem as ROM files. This way, a plugin can carry along
multiple components and data that form a complete Genode subsystem.
DDE Kit
=======
The DDE kit underwent slight modifications since the previous release.
It now provides 64-bit integer types and a revised virtual PCI bus
implementation.
Device drivers
##############
PCI bus
=======
Genode was tested on several hardware platforms in preparation of the
current release. This revealed some deficiencies with the PCI bus
driver implementation. The revised driver now efficiently supports
platforms with many PCI busses (as PCIe demands) and correctly handles
multi-function devices.
VESA framebuffer
================
We updated the configuration syntax of the VESA driver to better match
the style of new init syntax, preferring the use of attributes rather than
XML sub nodes. Please refer to the updated documentation at
'os/src/drivers/framebuffer/vesa/README'.
:Buffered output:
To accommodate framebuffer clients that need to read from the frame buffer,
in particular the nitpicker GUI server operating with alpha channels, we
introduced a buffered mode to the VESA driver. If enabled, the VESA driver
will hand out a plain memory dataspace to the client rather than the
physical framebuffer. Each time, the client issues as 'refresh' operation
on the framebuffer-session interface, the VESA driver copies the corresponding
screen region from the client-side virtual framebuffer to the physical
framebuffer. Note that the VESA driver will require additional RAM quota
to allocate the client buffer. If the quota is insufficient, the driver will
fall back to non-buffered output.
:Preinitialized video modes:
As an alternative to letting the VESA driver set up a screen mode, the
driver has become able to reuse an already initialized mode, which is useful
if the VESA mode is already initialized by the boot loader. If the screen
is initialized that way, the 'preinit' attribute of the 'config' node can
be set to '"yes"' to prevent the driver from changing the mode. This way,
the driver will just query the current mode and make the already
initialized framebuffer available to its client.
Audio
=====
We observed certain hardware platforms (in particular VirtualBox) to
behave strangely after ALSA buffer-underrun conditions. It seems that the
VirtualBox audio driver plays actually more frames than requested by
ALSA's 'writei' function, resulting in recurring replay of data that
was in the buffer at underrun time. As a work-around for this problem,
we zero-out the sound-hardware buffer in the condition of an ALSA buffer
underrun. This way, the recurring replay is still there, but it is
replaying silence.
To improve the support for sporadic audio output, we added a check for the PCM
state for buffer underruns prior issuing the actual playback. In the event of
an underrun, we re-prepare the sound card before starting the playback.
Furthermore, we implemented the new flush and channel-synchronization
abilities of the 'Audio_out_session' interface for the DDE Linux driver.
Paravirtualized Linux
#####################
To support the demo scenarios that showcase the paravirtualized Linux kernel,
we enhanced our custom stub drivers of the OKLinux kernel. Thereby, we have
reached a high level of integration of OKLinux with native Genode services,
including audio output, block devices, framebuffer output, seamless integration
with the Nitpicker GUI, and networking. All stub drivers are compiled in by
default and are ready to use by specifying a device configuration in the config
node for the Linux kernel. This way, one Linux kernel image can be easily used
in different scenarios.
:Integration with the Nitpicker GUI:
We enhanced our fbdev stub driver with a mechanism to merge view reposition
events. If a X11 window is moved, a lot of subsequent events of this type are
generated. Using the new optimization, only the most recent state gets
reported to Nitpicker, making the X11 GUI more responsive.
:UnionFS:
As we noticed that unionfs is required by all our Linux scenarios, we decided
to include and enable the patch by default.
:Network support:
With the introduction of the 'nic_bridge', multiple networking stacks can run
on Genode at the same time, which paves the way for new use cases. We have now
added a stub driver using Genode's 'Nic_session' interface to make the new
facility available to Linux.
:Audio output:
We adapted the ALSA stub driver to the changes of the 'Audio_out_session'
interface, using the new channel synchronization and flush functions.
Thereby, we optimized the stub driver to keep latency and seek times of
Linux userland applications reasonably low.
:Removed ROM file driver:
With the addition of the 'Block_session' stub driver, the original ROM file
driver is no longer required. So we removed the stub. For using ROM files as
disk images for Linux, there is the 'rom_loopdev' server, which provides a
block session that operates on a ROM file.
:Asynchronous block interface:
To improve performance, we changed the block stub driver to facilitate the
asynchronous mode of operation as provided by the 'Block_session' interface.
This way, multiple block requests can be issued at once, thereby shadowing
the round trip times for individual requests.
Protocol stacks and libraries
#############################
Gallium3D / Intel GEM
=====================
We improved the cache handling of our DRM emulation code (implementing
'drm_clflush_pages') and our EGL driver, thereby fixing caching
artifacts on i945 GPUs. Furthermore, we added a temporary work-around
for the currently dysfunctional sequence-number tracking with i945 GPUs.
On this chipset, issuing the 'MI_STORE_DWORD_INDEX' GPU command used
for tracking sequence numbers apparently halts the processing the command
stream. This condition is normally handled by an interrupt. However,
we have not enabled interrupts yet.
To prepare the future support for more Gallium drivers than i915, we
implemented a driver-selection facility in the EGL driver. The code
scans the PCI bus for a supported GPU and returns the name of the
corresponding driver library. If no driver library could be found,
the EGL driver falls back to softpipe rendering.
lwIP
====
We revised our port of the lwIP TCP/IP stack, and thereby improved its
stability and performance.
* The lwIP library is now built as shared object, following the convention
for libraries contained in the 'libports' repository.
* By default (when using the 'libc_lwip_nic_dhcp' library), lwIP will
issue a DHCP request at startup. If this request times out, the loopback
device is set as default.
* If there is no 'Nic' service available, the lwIP stack will fall back to
the loopback device.
* We increased the default number of PCBs in lwIP to 64.
* We removed a corner case of the timed semaphore that could occur
when a timeout was triggered at the same time ,'up' was called.
In this case, the semaphore was unblocked but the timeout condition
was not reflected at the caller of 'down'. However, the lwIP code
relies on detecting those timeouts.
Qt4
====
We implemented a custom *nitpicker plugin widget*, which allows for the
seamless integration of arbitrary nitpicker clients into a Qt4 application.
The primary use case is the browser plugin mechanism presented at
the Live CD. In principle, the 'QNitpickerViewWidget' allows for creating
mash-up allocations consisting of multiple native Genode programs. As shown
by the browser plugin demo, a Qt4 application can even integrate other
programs that run isolated from the Qt4 application, and thereby depend on
on a significantly less complex trusted computing base than the Qt4
application itself.
[image nitpicker_plugin]
The image above illustrates the use of the 'QNitpickerViewWidget' in the
scenario presented on the Live CD. The browser obtains the Nitpicker view to be
embedded into the website from the loader service, which virtualizes the
Nitpicker session interface for the loaded plugin subsystem. The browser then
tells the loader about where to present the plugin view on screen. But it has
neither control over the plugin's execution nor can it observe any user
interaction with the plugin.
New Gems repository with HTTP-based block server
################################################
To give the web-browser demo of our Live CD a special twist, and to show off
the possibilities of a real multi-server OS, we decided to implement the
somewhat crazy idea of letting a Linux OS run on a disk image fetched at
runtime from a web server. This way, the Linux OS would start right away and
disk blocks would be streamed over the network as needed. Implementing this
idea was especially attractive because such a feature would be extremely hard
to implement on a classical OS but is a breeze to realize on Genode where all
device drivers and protocol stacks are running as distinct user-level
components. The following figure illustrates the idea:
[image http_block]
The block stub driver of the Linux kernel gets connected to a special block
driver called 'http_block', which does not access a real block device but
rather uses TCP/IP and HTTP to fetch disk blocks from a web server.
Because the 'http_block' server is both user of high-level functionality (the
lwIP stack) and provider of a low-level interface ('Block_session'), the
program does not fit well into one of the existing source-code repositories.
The 'os' repository, which is normally hosting servers for low-level interfaces
is the wrong place for 'http_block' because this program would make the 'os'
repository depend on the higher-level 'libports' repository where the 'lwip'
stack is located. On the other hand, placing 'http_block' into the 'libports'
repository is also wrong because the program is not a ported library. It merely
uses libraries provided by 'libports'. In the future, we expect that native
Genode components that use both low-level and high-level repositories will
become rather the norm than an exception. Therefore, we introduced a new
repository called 'gems' for hosting such programs.
Tools
#####
Automated coding-style checker
==============================
As Genode's code base grows and new developers start to get involved,
we noticed recurring questions regarding coding style. There is a
[http://genode.org/documentation/developer-resources/coding_style - document]
describing our coding style but for people just starting to get involved,
adhering all the rules can become tedious. However, we stress the importance
of a consistent coding style for the project. Not only does a consistent style
make the framework more approachable for users, but it also eases the work
of all regular developers, who can feel right at home at any part of
the code.
To avoid wasting precious developer time with coding-style fixes, we
have created a tool for the automated checking and (if possible) fixing
the adherence of source code to Genode's coding style. The tool is
located at 'tool/beautify'. It takes a source file as argument and
reports coding-style violations. The checks are fairly elaborative:
* Placement of braces and parenthesis
* Indentation and alignment, trailing spaces
* Vertical spacing (e.g., between member functions, above comments)
* Naming of member variables and functions (e.g., private members start with '_')
* Use of upper and lower case
* Presence of a file header with the mandatory fields
* Policy for function-header comments (comment at declaration, not
at implementation)
* Style of single-line comments, function-header comments, multi-line comments
The user of 'beautify' may opt to let the tool fix most of the violations
automatically by specifying the command line arguments '-fix' and '-write'.
With only the '-fix' argument, the tool will output the fixed version of
the code via stdout. By specifying the '-write' argument, the changes will
be written back to the original file. In any case, we strongly recommend
to manually inspect all changes made by the tool.
Under the hood, the tool consists of two parts. A custom C++ parser called
'parse_cxx' reads the source code and converts it to a syntax tree. In the
syntax tree, all formating information such as whitespaces are preserved.
The C++ parser is a separate command-line tool, which we also use for
other purposes (e.g., generating the API documentation at the website).
The actual 'beautify' tool calls 'parse_cxx', and applies its checks and
fixes to the output of 'parse_cxx'. For this reason, both tools have to
reside in the same directory.
Platform-specific changes
#########################
OKL4
====
:Added support for shared interrupts:
The Genode Live CD operates on a large number of devices that trigger
interrupts (USB, keyboard, mouse, ATAPI, timer, network). On most
platforms, the chances are extremely high that some of them use
the same IRQ line. Therefore, we enhanced core's IRQ service to
allow multiple clients to request the same IRQ. If the interrupt occurs,
all clients referring to this interrupt are notified. The interrupt
gets cleared after all of those clients responded. Even though, we regard
PIC interrupts as a legacy, the support of shared interrupts enables
us to use OKL4 with such complex usage scenarios.
:Revised page-fault handling:
If a page fault occurs, the OKL4 kernel delivers a message to the page-fault
handler. The message contains the page-fault address and type as well as the
space ID where the fault happened. However, the identity of the faulting
thread is not delivered. Instead, the sender ID of the page fault message
contains the KTCB index of the faulting thread, which is only meaningful
within the kernel. This KTCB index is used as a reply token for answering the
page fault message. We wondered about why OKL4 choose to deliver the KTCB
index rather then the global thread ID as done for plain IPC messages. The
only reasonable answer is that by using the KTCB index directly in OKL4's
page-fault protocol, one lookup from the userland-defined thread ID to the
KTCB index can be avoided. However, this comes at the cost of losing the
identity of the faulting thread. We used to take the space ID as a key for
the fault context within core. However, with Genode's user-level page-fault
mechanism, this simplification does not suffice anymore. We have to know the
faulting thread as a page fault may not be answered immediately but at a
later time. During that time, the page-fault state has to be stored at core's
representation of the faulting thread. Our solution is reverting OKL4's
page-fault protocol to operate with global thread IDs only and to never make
kernel-internal KTCB indices visible at the user land. You can find the patch
for the OKL4 kernel at 'base-okl4/patches/reply_tid.patch'.
:Reboot via kernel debugger:
We fixed the reboot code of OKL4's kernel debugger to improve our work
flow. The patch can be found at 'base-okl4/patches/kdb_reboot.patch'.
:Relieved conflict with libc 'limits.h':
For some reason, the OKL4 kernel bindings provide definitions
normally found in libc headers. This circumstance ultimately leads
to trouble when combining OKL4 with a real C runtime. We have
relieved the problem with the patch 'base-okl4/patches/char_bit.patch'.
:Exception handling:
We added a diagnostic message to core that reports about exceptions
such as division by zero.
Pistachio
=========
Our revised syscall bindings for supporting position-independent code
on L4ka::Pistachio have been integrated into the mainline development
of the kernel. Therefore, the patch is not needed anymore when using
a kernel revision newer than 'r791:0d25c1f65a3a'.
Linux
=====
On Linux, we let the kernel manage all virtual address spaces for us,
except for the thread-context area. Because the kernel does not know
about the special meaning of the thread-context area, it may choose to
use this part of the virtual address space as target for 'mmap'. This
may lead to memory corruption. Fortunately, there is a way to tell the
kernel about virtual address regions that should be reserved. The
trick is to pre-populate the said region with anonymous memory using
the 'mmap' arguments 'MAP_PRIVATE', 'MAP_FIXED', 'MAP_ANONYMOUS', and
'PROT_NONE'. The kernel will still accept a fixed-address mapping
within such a reserved region (overmap) but won't consider using the
region by itself. The reservation must be done at the startup of each
process and each time when detaching a dataspace from the thread
context area. For the process startup, we use the hook function
'main_thread_bootstrap' in 'src/platform/_main_helper.h'. For reverting
detached dataspaces to a reserved region within the context area, we
added as special case to 'src/base/env/rm_session_mmap.cc'.
For hybrid programs (Genode processes that link against native
shared libraries of the Linux system), which are loaded by the dynamic
linker of Linux, we must further prevent the dynamic linker from
populating the thread-context area. This is achieved by adding a
special program segment at the linking stage of all elf binaries.

876
doc/release_notes-11-02.txt
View File

@@ -1,876 +0,0 @@
===============================================
Release notes for the Genode OS Framework 11.02
===============================================
Genode Labs
One year ago, the release 10.02 was our break-through with regard to the support
of multiple kernels as base platform for Genode. With the added support for
the NOVA hypervisor and the Codezero kernel, Genode applications could be executed
on 6 different kernels. With the current release, we take our commitment to
kernel platform support even further. With the added support for the Fiasco.OC
kernel, we make Genode available on one of the most feature-rich modern microkernels.
Additionally, we entered the realms of kernel design with our new platform support
for the Xilinx MicroBlaze architecture. This platform support comes in the shape
of a custom kernel specifically targeted to the MicroBlaze CPU architecture.
Furthermore, we updated our support for the NOVA Hypervisor to the bleeding-edge
version 0.3, which has been released earlier this month.
With the current support for 8 different kernel platforms (L4/Fiasco, Linux,
L4ka::Pistachio, OKL4, NOVA, Codezero, Fiasco.OC, and native MicroBlaze), testing
and integrating application scenarios across all platforms becomes increasingly
challenging. Therefore, we introduce a new framework for automating such tasks.
Thanks to the tight integration of the automation tool with Genode's build system,
going back and forth between different kernels becomes an almost seamless
experience.
Functionality-wise, the release carries on our vision to create a highly secure
yet easy to use general-purpose operating system. Because the Genode framework
is developed on Linux using the wonderful GNU tools, we consider the
availability of the GNU user land on Genode as crucial for using the system by
ourself. This motivation drives the creation of a custom execution environment
for GNU software on top of Genode. With the current release, we are proud to
present the first pieces of this execution environment. Even though not fully
functional yet, it clearly shows the direction of where we are heading.
Support for Fiasco.OC
#####################
The OC in the name of the Fiasco.OC kernel stands for "object capability", hinting
at the most significant feature that sets current-generation microkernels such as
NOVA, seL4, and Fiasco.OC apart from their predecessors. Whereas previous L4 kernels
succeeded in protecting subsystems from each other, the new generation of kernels
is geared towards strict security policies. Traditionally, two protection domains
were able to communicate with each other if they both agreed. Communication partners
were typically globally known via their respective thread/task IDs. Obviously, this
policy is not able to guarantee the separation of subsystems. If two subsystems
conspire, they could always share information. Object-capability-based kernels
are taking the separation much further by prohibiting any communication between
protection domains by default. Two protection domains can communicate only if
a common acquaintance of both agrees. This default-deny policy facilitates the
creation of least-privilege security policies. From the ground up, Genode has
been designed as a capability-based system which is naturally capable of leveraging
kernel-based object-capability support if present. After NOVA, Fiasc.OC is the
second of Genode's base platforms that provides this feature.
Apart from being a capability-based kernel, Fiasco.OC has a number of compelling
features such as thorough support for ARM platforms and the x86 32/64 bit
architectures. It supports SMP, hardware virtualization, and provides special
optimizations for running paravirtualized operating systems.
Technically, Fiasco.OC is the successor of the L4/Fiasco kernel developed by
the OS group of the TU-Dresden. However, the kernel interface of Fiasco.OC has
not much in common with L4/Fiasco. Some heritages are still there (e.g., IPC
timeouts) but the kernel API has evolved to a fully object-oriented model.
:Thanks:
We are indebted to the main developer of Fiasco.OC Alexander Warg for being
very reponsive to our inquiries while doing the porting work. Thanks to his
support, the adaptation of Genode to this kernel has been an almost smooth
ride.
Prerequisites
=============
You need GNU C & C++ Compilers, GNU Binutils, GNU Make, and Perl to use the
Fiasco.OC build system. On Debian/Ubuntu systems, you have to install the
following packages:
! apt-get install make gawk g++ binutils pkg-config subversion
Moreover, you need to download and install the tool-chain used by Genode. Have
a look at this page:
:[http://genode.org/download/tool-chain]:
Genode tool-chain
Downloading and building Fiasco.OC
==================================
Checkout the Fiasco.OC sources and tool-chain to an appropriated directory:
! export REPOMGR_SVN_REV=27
! svn cat http://svn.tudos.org/repos/oc/tudos/trunk/repomgr |\
! perl - init http://svn.tudos.org/repos/oc/tudos fiasco l4re
Building the kernel
~~~~~~~~~~~~~~~~~~~
Create the build directory for the kernel:
! cd <path_to_fiasco_src_dir>/src/kernel/fiasco
! make BUILDDIR=<path_to_kernel_build_dir>
Go to the build directory, configure the kernel:
! cd mybuild
! make config
This will launch the configuration menu. Here you can configure your kernel.
The default config is just fine to test the Genode port. It will build a
uniprocessor IA32 kernel with debugging features enabled. You can exit the menu and
save the configuration by simply typing 'x'.
Now, build Fiasco.OC by invoking:
! make
Building necessary tools
~~~~~~~~~~~~~~~~~~~~~~~~
To practically use Fiasco.OC, you need in addition to the kernel a tool to
bootstrap it, and the initial pager of the system, namely 'sigma0'. Both tools
can be found in the L4 runtime environment's base directory. Outgoing from
the directory where you checked out the sources, you have to change to the
following directory:
! cd <path_to_fiasco_src_dir>/src/l4
Create another build directory:
! make B=<path_to_l4re_build_dir>
Again, you might want to tweak the configuration:
! make O=<path_to_l4re_build_dir> config
Finally, build the tools:
! make O=<path_to_l4re_build_dir>
Building the Fiasco.OC version of Genode
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The Fiasco.OC version of Genode is available at the Genode public subversion repository:
:http://genode.org/download/subversion-repository:
Information about accessing the Genode public subversion repository
Go to a directory where you want the Genode/Fiasco.OC build directory to remain. Use
the helper script in the 'tool/builddir' directory of the Genode source tree to
create the initial build environment. You need to state the absolute path to the
build directory of the L4 runtime environment as 'L4_DIR', as it contains the kernel
bindings needed by the Genode port.
! <path_to_genode_src_dir>/tool/builddir/create_builddir foc_x86_32 \
! L4_DIR=<path_to_l4re_build_dir> \
! GENODE_DIR=<path_to_genode_src_dir> \
! BUILD_DIR=<path_to_genode_build_dir>
Now, go to the newly created build directory and type make.
! cd <path_to_genode_build_dir>
! make
Booting Genode on top of Fiasco.OC
==================================
Example GRUB configuration entry:
! timeout 0
! default 0
!
! title Genode on Fiasco.OC
! kernel /bootstrap -modaddr=0x01100000
! module /fiasco -serial_esc
! module /sigma0
! module /core
! module /init
! module /config
! module /pci_drv
! module /vesa_drv
! module /ps2_drv
! module /timer
! module /nitpicker
! module /launchpad
! module /liquid_fb
! module /scout
! module /testnit
! module /nitlog
For an example of a matching Genode 'config' file, please take a look
at 'os/config/demo'.
The Genode binaries are located in '<path_to_genode_build_dir>/bin',
the 'fiasco' kernel in '<path_to_kernel_build_dir>'. Assuming you compiled
for x86/586 (the default), you can find the 'bootstrap' binary in
'bin/x86_586' and 'sigma0' in 'bin/x86_586/l4f' within the
'<path_to_l4re_build_dir>' directory.
Current state
=============
The adaptation of Genode to Fiasco.OC covers most parts of the Genode API
including advanced semantics such as cancelable locks and support for
real-time priorities. So far, it has been tested on the x86 architecture.
Because 'base-foc' does not contain x86-specific code, we expect no major
roadblocks for running Genode on Fiasco.OC on ARM. However, we have not
exercised tests in this regard.
As of today, there exist the following limitations of the Fiasco.OC support:
* The dynamic linker is not yet adapted to Fiasco.OC. Special care must
be taken for handling the parent capability for dynamically loaded
programs. We have already covered this issue for the NOVA version but
the adaptation to Fiasco.OC remains yet to be done.
* The destruction of sub systems is not yet fully stable. Because Genode
forms a more dynamic workload than the original userland accompanied with
the kernel, the usage pattern of the kernel API triggers different
effects. We are working with the Fiasco.OC developers to remedy this
issue.
* The signalling framework is not yet supported. A design exist but it is
not implemented yet.
We believe however that none of these limitations are a significant hurdle for
starting to use Genode with this kernel. Please expect this issues to be
resolved with the upcoming Genode release.
Technical details about 'base-foc'
==================================
The following technical bits are worth noting when exploring the use of
Genode with the 'base-foc' platform.
* The timer implementation uses a one thread-per-client mode of operation.
We use IPC timeouts as time source. Hence, the timer driver is hardware
independent and should work out of the box on all hardware platforms
supported by Fiasco.OC.
* Each 'Server_object' of Genode corresponds to a so-called IPC gate,
which is the Fiasco.OC kernel object used for capability invocation.
Therefore, protection and object integrity is provided at the fine
granularity of single 'Server_objects'. This is in line with our
support for NOVA's implementation of capability-based security.
* In contrast to the lock implementation that we used with the original
L4/Fiasco kernel, the 'base-foc' lock is a fully-featured Genode lock
with support for lock cancellation and blocking. For blocking and
waking up lock applicants, we use Fiasco.OC's IRQ objects.
* The allocator used for managing process-local capability selectors
does not yet support the reuse of capability selectors.
Further Information
===================
:genode/tool/builddir/README:
Reference manual for the 'create_builddir' script
:[http://os.inf.tu-dresden.de/fiasco]:
Official website for the Fiasco.OC microkernel.
Noux - an execution environment for the GNU userland
####################################################
Even though Genode is currently mainly geared to the classical special-purpose
application domains for microkernel-based systems, the main property that sets
Genode apart from traditional systems is the thorough support for dynamic
workloads and the powerful mechanisms for handling hardware resources and
security policies in highly dynamic setting. We are convinced that Genode's
architecture scales far beyond static special-purpose domains and believe in
the feasibility of Genode as a solid foundation for a fully-fledged general
purpose operating system. Internally at Genode Labs, we set up the ultimate
goal to switch from Linux to Genode for our day-to-day work. We identified
several functionalities that we could not live without and systematically try
to bring those features to Genode. Of course, the most fundamental programs
are the tools needed to develop and build Genode. Currently we are developing
on Linux and enjoy using the GNU userland.
Consequently, we require a solution for using this rich tool set on Genode.
The straight-forward way for making these tools available on Genode would be
running them within a virtualized Linux instance (e.g., using OKLinux on OKL4).
However, this approach would defeat our actual goal to create a highly secure
yet easy to use working environment because adding Linux to the picture would
involve administering the virtualized Linux system. We would prefer a native
solution that makes the overall system less, not more, complicated. This way
the idea for a native execution environment for the GNU userland on Genode
was born. The implementation is called Noux and the first bits of code are
featured in the 'ports' repository. Noux consists of two parts, a build
environment for compiling GNU programs such that they can be run as Genode
processes and an execution environment that provides the classical UNIX
functionality to these programs.
Noux build environment
======================
From our experience, porting existing UNIX applications to a non-UNIX system
tends to be a task of manual and time-consuming labour. One has to loosely
understand the build system and the relationship of the involved source codes,
implement dummy functions for unresolved references, and develop custom glue
code that interfaces the ported application to the actual system. Taking the
shortcut of changing the original code has to be avoided at any cost because
this produces recurring costs in the future when updating the application. In
short, this long-winding process does not scale. For porting a tool set such as
the GNU userland consisting of far more than a three-digit number of individual
programs, this manual approach becomes unfeasible. Therefore, we have created
a build environment that facilitates the use of the original procedure of
invoking './configure && make'. The challenge is to supply configure with
the right arguments and environment variables ('CFLAGS' and the like) such that
the package is configured against the Genode environment. The following
considerations must be taken:
* Configure must not detect any global headers (e.g., '/usr/include/')
or libraries (e.g., '/usr/lib/'). This can be achieved by the '-nostdinc' and
'-nostdlib' options
* Configure has to use the same include-search paths as used for compiling
normal libc-using Genode programs
* Configure must use the Genode tool chain
* The final linking stage must use the Genode linker script, the Genode
base libraries, and other Genode-specific linker arguments.
Thanks to the power of the GNU build system, all this can be achieved by
supplying arguments to './configure' and indirectly to the 'make' process via
environment variables. The new Noux build environment takes care of these
precautions. It comes in the form of the 'ports/mk/noux.mk' file which enables
the seamless integration of GNU packages with the Genode build system. To
compile a GNU package, the manual steps needed are reduced to the creation of a
'target.mk' file representing the package. This 'target.mk' defines the name
of the package (by default, the basename of the 'target.mk' enclosing directory
is assumed) and the location of the source package. With this approach, we
managed to build 'coreutils' (over 100 small UNIX utilities such as 'ls', 'cp',
'sort'), 'binutils' (GNU linker, assembler, object-file tools), 'findutils'
('find', 'xargs'), 'bash', 'dash', GNU make, and finally the GNU compiler
collection including 'g++'. The resulting binaries are ready to be executed as
native Genode processes. However, without the right environment that presents
the program the needed UNIX functionality, those programs won't do much.
This leads us to the Noux execution environment.
Noux execution environment
==========================
The Noux execution environment plays the role of a UNIX kernel for programs
built via the Noux build environment. In contrast to a real kernel, the Noux
environment is a plain Genode user-level process that plays the role of being
the parent of one or multiple Noux processes. In addition of providing the
'Genode::Parent' interface, Noux also provides a locally implemented service called
'Noux::Session' that offers UNIX-like system-calls via an RPC interface. Each
hosted program is linked against a special Noux libc plugin that catches all
libc calls that would normally result in a system call. It then transparently
forwards this function call to the 'Noux::Session' interface.
Currently the Noux execution environment implements the following
system calls: 'getcwd', 'write', 'stat', 'fstat', 'fcntl', 'open',
'close', 'dirent', 'fchdir', 'read', and 'execve'.
The execution environment submits arguments (argc, argv, environment) to the
hosted program, manages its current working directory and receives its exit
code. File operations are targeted to a custom VFS infrastructure, which
principally allows a flexible configuration of the virtual file system visible
to the hosted programs. At the current stage, Noux supports mounting plain tar
archives obtained from core's ROM service as read-only file system. On startup,
the Noux environment starts one process (the init process) and connects the
file descriptor 1 (stdout) to Genode's LOG service.
State of the implementation
~~~~~~~~~~~~~~~~~~~~~~~~~~~
The infrastructure implemented so far already allows the execution of many simple
UNIX tools such as 'ls -lRa', 'echo', 'seq', 'find'. The 'execve' system call
is implemented such that a new process is started that inherits the file
descriptors and the PID of the calling process. This allows using the exec
functionality of the 'bash' shell. However, because 'fork' is not implemented
yet, there is currently no way to start multiple programs hosted in a single
Noux execution environment.
As of today, the Noux environment is not considered to be usable for practical
purposes. However, it clearly shows the feasibility of the path we are walking.
With the foundation laid, we are looking forward to expanding Noux to a capable
solution for running our beloved GNU userland tools on Genode.
Vision
~~~~~~
The most significant intermediate result of pursuing the development of Noux is
the realization that such an environment is not exceedingly complex. Because of
the combination with Genode, we only need to provide a comfortable runtime as
expected by user processes but we can leave much of intricate parts of UNIX out
of the picture. For example, because we handle device drivers on Genode, we do
not need to consider device-user interaction in Noux. As another example,
because the problem of bootstrapping the OS is already solved by Genode, there
is no need to run an 'init' process within Noux. Our vision foresees that Noux
runtimes are to be created on demand for individual tasks such as editing a
file (starting a custom Noux instance containing only the file to edit and the
text editor), compiling source code (starting a custom Noux instance with only
the source code and the build tools). Because Noux is so simple, we expect the
runtime overhead of starting a Noux instance to be not more than the time
needed to spawn a shell in a normal UNIX-like system.
Test drive
~~~~~~~~~~
To give Noux a spin, we recommend using Linux as base platform as this is
the platform we use for developing it. First, you will need to download the
source code of the GNU packages. From within the 'ports' repository,
use the following command:
! make prepare PKG=coreutils
This command will download the source code of the GNU coreutils. You may
also like to give the other packages a try. To see what is available,
just call 'make' without any argument.
Create a build directory (e.g., using tool/builddir/create_builddir).
Change to the build directory and issue the command
! make run/noux
This command will execute the run script provided at 'ports/run/noux.run'.
First it builds core, init, and coreutils. Then it creates a tar archive
containing the installed coreutils. Finally, it starts the Noux environment on
Genode. Noux then mounts the TAR archive as file system and executes 'ls -laR',
showing the directory tree.
Approaching platform support for Xilinx MicroBlaze
##################################################
With the release 11.02, we are excited to include the first version of our
custom platform support for the Xilinx MicroBlaze CPU architecture. MicroBlaze
is a so-called softcore CPU, which is commonly used as part of FPGA-based
System-on-Chip designs. At Genode Labs, we are regularly using this IP core,
in particular for our Genode FPGA Graphics Project, which is a GUI software stack
and a set of IP cores for implementing fully-fledged windowed GUIs on FPGAs:
:Website of the Genode FPGA Graphics Project:
[http://genode-labs.com/products/fpga-graphics]
Ever since we first released the Genode FPGA project, we envisioned to combine
it with the Genode OS Framework. In Spring 2010, Martin Stein joined our team
at Genode Labs and accepted the challenge to bring the Genode OS Framework to
the realms of FPGA-based SoCs. Technically, this implies porting the framework
to the MicroBlaze CPU architecture. In contrast to most softcore CPUs such as
the popular Lattice Mico32, the MicroBlaze features a MMU, which is a fundamental
requirement for implementing a microkernel-based system. Architecturally-wise
MicroBlaze is a RISC CPU similar to MIPS. Many system parameters of the CPU
(caches, certain arithmetic and shift instructions) can be parametrized at
synthesizing time of the SoC. We found that the relatively simple architecture
of this CPU provides a perfect playground for pursuing some of our ideas about
kernel design that go beyond the scope of current microkernels. So instead of
adding MicroBlaze support into one of the existing microkernels already
supported by Genode, we went for a new kernel design. Deviating from the typical
microkernel, which is a self-sufficient program running in kernel mode that
executes user-level processes on top, our design regards the kernel as a part of
Genode's core. It is not a separate program but a library that implements the
glue between user-level core and the raw CPU. Specifically, it provides the
entrypoint for hardware exceptions, a thread scheduler, an IPC mechanism, and
functions to manipulate virtual address spaces (loading and flushing entries
from the CPU's software-loaded TLB). It does not manage any physical memory
resources or the relationship between processes. This is the job of core.
From the kernel-developer's point of view, the kernel part can be summarized as
follows:
* The kernel provides user-level threads that are scheduled in a round-robin
fashion.
* Threads can communicate via synchronous IPC.
* There is a mechanism for blocking and waking up threads. This mechanism
can be used by Genode to implement locking as well as asynchronous
inter-process communication.
* There is a single kernel thread, which never blocks in the kernel code paths.
So the kernel acts as a state machine. Naturally, there is no concurrency in the
execution paths traversed in kernel mode, vastly simplifying these code parts.
However, all code paths are extremely short and bounded with regard to
execution time. Hence, we expect the interference with interrupt latencies
to be low.
* The IPC operation transfers payload between UTCBs only. Each thread has a
so-called user-level thread control block which is mapped transparently by
the kernel. Because of this mapping, user-level page faults cannot occur
during IPC transfers.
* There is no mapping database. Virtual address spaces are manipulated by
loading and flushing physical TLB entries. There is no caching of mappings
done in the kernel. All higher-level information about the interrelationship
of memory and processes is managed by the user-level core.
* Core runs in user mode, mapped 1-to-1 from the physical address space
except for its virtual thread-context area.
* The kernel paths are executed in physical address space (MicroBlaze).
Because both kernel code and user-level core code are observing the same
address-space layout, both worlds appear to run within a single address
space.
* User processes can use the entire virtual address space (4G) except for a
helper page for invoking syscalls and a page containing atomic operations.
There is no reservation used for the kernel.
* The MicroBlaze architecture lacks an atomic compare-and-swap instruction. On
user-level, this functionality is emulated via delayed preemption. A kernel-
provided page holds the sequence of operations to be executed atomically and
prevents (actually delays) the preemption of a thread that is currently
executing instructions at that page.
* The MicroBlaze MMU supports several different page sizes (1K up to 16MB).
Genode fully supports this feature for page sizes >= 4K. This way, the TLB
footprint can be minimized by choosing sensible alignments of memory
objects.
Current state
=============
The MicroBlaze platform support resides in the 'base-mb' repository. At the
current stage, core is able to successfully start multiple nested instances of
the init process. Most of the critical kernel functionality is working. This
includes inter-process communication, address-space creation, multi-threading,
thread synchronization, page-fault handling, and TLB eviction.
This simple scenario already illustrates the vast advantage of
using different page sizes supported by the MicroBlaze CPU. If using
4KB pages only, a scenario with three nested init processes produces more than
300.000 page faults. There is an extremely high pressure on the TLB, which
only contains 64 entries. Those entries are constantly evicted so that
threshing effects are likely to occur. By making use of flexible page
sizes (4K, 16K, 64K, 256K, 1M, 4M, 16M), the number of page faults gets
slashed to only 1.800, speeding up the boot time by factor 10.
Currently, there is no restriction of IPC communication rights. Threads are
addressed using their global thread IDs (in fact, using their respective
indices in the KTCB array). For the future, we are planning to add
capabilty-based delegation of communication rights.
Building and using Genode on MicroBlaze
=======================================
For building Genode for the MicroBlaze platform, you need the MicroBlaze
tool chain as it comes with the Xilinx EDK. The tool chain is typically
prefixed with 'mb-'. Please make sure that the tool chain's 'bin/' directory
is included in your 'PATH' environment variable.
For building and starting Genode on MicroBlaze, you first need to create
a build directory using the build-directory creation tool:
! tool/builddir/create_builddir microblaze \
! BUILD_DIR=</path/to/build/dir> \
! GENODE_DIR=</path/to/genode/dir>
The 'base-mb' repository comes with support for Genode's run tool. In order to
use it, you will first need to declare the location of your qemu binary using
the 'QEMU=/path/to/qemu' variable in the '<build-dir>/etc/microblaze.conf'
file. Then you will be able to start an example scenario by issuing the
following command from within your build directory:
! make run/nested_init
Thereby, the 'run' tool will attempt to start core using the microblaze version
of qemu.
You can also find a simple hello-world example at 'base-mb/src/test/hello'.
The corresponding run script is located at 'base-mb/run/hello.run'. You can
execute it via 'make run/hello' from the build directory.
Note that currently, all boot modules are linked against the core binary.
To change the boot modules, the file 'base-mb/src/core/boot_modules.s' must
be modified.
For reference, we are using the following tools:
* mb-g++ (GCC) 4.1.1 20060524 (Xilinx 11.2 Build EDK_LS2.2
20 Apr 2009 Xilinx 11.2 Build EDK_LS2.2 23 Apr 2009)
* GNU ld version 2.16 Xilinx 11.2 Build EDK_LS2.2 23 Apr 2009
* GNU assembler 2.16 Xilinx 11.2 Build EDK_LS2.2 23 Apr 2009
* QEMU emulator version 0.14.50, Copyright (c) 2003-2008 Fabrice Bellard
Petalogix linux reference design targeting Xilinx Spartan 3ADSP-1800 boards.
Supporting the NOVA hypervisor version 0.3
##########################################
NOVA is a so called microhypervisor - a modern capability-based microkernel
with special support for hardware-based virtualization and IOMMUs. Since we
incorporated the initial support for the NOVA hypervisor in Genode one year
ago, this kernel underwent multiple revisions. The latest version was released
earlier this month. To our delight, much of the features that we missed from
the initial release had been implemented during the course of the last year. We
are especially happy about the fully functional 'revoke' system call and the
support for remote kernel-object creation.
With the Genode release 11.02, we officially support the latest NOVA version.
The update of Genode to the new version required two steps. First, because many
details of the kernel interface were changed between version 0.1 and version
0.3, we had to revisit our syscall bindings and adapting our code to changed
kernel semantics. Second, we filled our 'base-nova' code related to object
destruction and unmapping with life to benefit from NOVA's 'revoke' system
call. Consequently, we are now able to run the complete Genode software stack
including the dynamic linker on NOVA.
Note that for using Genode on NOVA, you will need to apply a small patch to the
NOVA source code. This patch enables the re-use of user-level thread control
blocks in the kernel. The patch can be found at 'base-nova/patches/utcb.patch'.
When executing NOVA on qemu, please specify the '-cpu coreduo' argument to the
qemu command line. When using Genode 'run' tool, you may assign this argument
to the 'QEMU_OPT' variable in '<build-dir>/etc/build.conf'.
:Thanks:
We are grateful for the ongoing very pleasant collaboration with Udo Steinberg
who is the driving force behind NOVA. Thanks for the ultra-fast responses to our
questions and for considering our suggestions regarding the feature set of
NOVA's kernel interface!
Base framework
##############
Upgrading existing sessions
===========================
Genode enables a client of a service to lend parts of its own resources to
the service when opening a session. This way, servers do not need to allocate
own resources on behalf of their clients and become inherently robust against
resource-exhaustion-based denial-of-service attacks.
However, there are cases when the client can not decide about the amount of
resources to lend at session-creation time. In such cases, we used to devise an
overly generous client policy. Now, we have added a new 'upgrade' function to
the 'Parent' and 'Root' interfaces that enables a client to upgrade the
resources of an existing session.
For the 'env()->rm_session()' and 'env()->ram_session()' of processes using
the Genode 'env' library, we implemented a transparent quota upgrade that kicks in
in the event of an exceeded metadata backing store.
Comprehensive accounting of core resources
==========================================
We changed all services of core to limit their respective resource usage
specifically for each individual session. For example, the number of dataspaces
that can be handled by a particular region-manager (RM) session depends on the
resource donation attached to the session. To implement this accounting scheme
throughout core, we added a generic 'Allocator_guard' utility to
'base/include/'. We recommend using this utility when implementing resource
multiplexers, in particular multi-level services. Thanks to this change in
core, the need for a slack memory reservation in core has vanished.
Various changes
===============
The remaining parts of the base API underwent no fundamental revision. The
changes are summarized as follows.
:C++ Support:
We removed 'libgcc' from our C++ support library ('cxx') and link
it to each individual final target and shared library instead. This change alleviates
the need to abuse the 'KEEP_SYMBOLS' mechanism that we used in 'cxx' to
keep libc-dependencies of GCC's support libraries local to the 'cxx'
library. Besides the benefit of reducing heuristics, this change improves
the compatibility with recent cross-compiling tool chains.
Furthermore, we added 'realloc' to the local libc support of the 'cxx'
library because recent ARM tool chains tend to use this function.
:Argument handling for 'main()':
We added a hook to the startup code to enable the implementation of
custom facilities for passing arguments to the main function. The
hook uses the global variables 'genode_argc' and 'genode_argv'.
:Child-exit policy hook:
We enhanced the 'Child_policy' with a new policy interface that allows
a simplified implementation of policies related to program termination.
:Changed API of 'Range_allocator':
We changed the return value of 'alloc_addr' to distinguish different error
conditions. Note that the boolean meaning of the return value is inverted.
Please check your uses of 'alloc_addr'!
Operating-system services and libraries
#######################################
C Runtime
=========
In conjunction with our work on Noux, we improved Genode's C runtime at many
places. First, we added libstdtime and some previously missing bits of libgdtoa
to the libc. These additions largely alleviate the need for dummy stubs, in
particular time-related functions. Second, we added the following functions to
our libc plugin interface: 'dup2', 'fchdir', 'fcntl', 'fstat', 'stat', and
'write'. This enables the creation of advanced libc plugins simulating a whole
file system as done with Noux. Still, there are a number of dummy stubs found
at 'libc/src/lib/libc/dummy.cc'. However, those stubs are now all defined as
weak symbols such that they can be overridden by libc plugins. Finally, we have
replaced the original 'exit' implementation that comes with the libc with a
Genode-specific version. The new version reports the exit code of the
application to the parent process via an 'Parent::exit()' RPC call.
Until now, Genode's libc magically handled output to stdout and stderr by
printing messages via Genode's LOG interface. We have now replaced this
hard-wired interface by an optional libc plugin called 'libc_log'. If present, write
operations to stdout are caught at the libc plugin interface and delegated to
the plugin, which implements the output to the LOG interface. If you have an
application using Genode's libc, you might consider adding the 'libc_log'
library to your 'target.mk' file.
Support for big numbers by the means of libgmp and libmpfr
==========================================================
We have now include both the GNU Multiple Precision Arithmetic Library and
(GMP) and MPFR to the 'ports' repository. This work was specifically motivated
by our port of GCC to Genode as GCC version 4.4.5 requires both libraries.
Because we intend to use those libraries primarily on x86_32, the current port
covers only this architecture. However, expanding the port to
further CPU architectures should be straight-forward if needed.
Furthermore, you can now also find GCC's 'longlong.h' header at
'libports/include/gcc'.
Qt4 updated to version 4.7.1
############################
The current release bumps the supported Qt4 version from 4.6.2 to 4.7.1 and the
Arora web browser (located at the ports repository) from version 0.10.2 to
version 0.11. Of course, we updated our custom additions such as our custom
Nitpicker plugin widget that enables the seamless integration of native
Nitpicker GUI clients into Qt4 applications to work with the new Qt4 version.
Tools
#####
Tool chain update to GCC 4.4.5 and Binutils 2.21
================================================
We upgraded the official Genode tool chain from gcc 4.2.4 to gcc 4.4.5. Please
update your tool chain by downloading the new binary archive (available for x86_32)
or building the tool chain from source using our 'tool/tool_chain' utility.
New support for automated integration and testing
=================================================
With the growing number of supported base platforms, the integration and testing
of Genode application scenarios across all kernels becomes
increasingly challenging. Each kernel has a different boot mechanism and
specific requirements such as the module order of multiboot modules (Fiasco's
bootstrap, Pistachio's sigma0 and kickstart), kernel parameters, or the
invocation of a single-image creation tool (OKL4's elfweaver). To make our
life supporting all those platforms easier, we have created a tool called
'run', which is tightly integrated within Genode's build system. In short 'run'
gathers the intrinsics in the form of a 'run/env' file specific for the
platform used by the current build directory from the respective
'base-<platform>' repository. It then executes a so-called run script, which
contains all steps needed to configure, build, and integrate an application
scenario. For example, a typical run script for building and running a test
case resides in a file called '<any-repository>/run/<run-script-name>.run' and
looks as follows:
! build "core init test/exception"
! create_boot_directory
! install_config {
! <config>
! <parent-provides>
! <!--<service name="ROM"/>-->
! <service name="LOG"/>
! </parent-provides>
! <default-route>
! <any-service> <parent/> </any-service>
! </default-route>
! <start name="test-exception">
! <resource name="RAM" quantum="1M"/>
! </start>
! </config>
! }
! build_boot_image "core init test-exception"
! append qemu_args "-nographic -m 64"
! run_genode_until {.*Exception \(label 0xffb0\) occured.*} 10
First, the build system is instructed to create the targets specified as argument
for the 'build' function. Next, for the integration part, a new boot directory is
created. On most kernel platform, the respective location of the boot directory
is '<build-dir>/var/run/<run-script-name>'. Initially, this directory is empty.
It gets populated with a 'config' file specified as argument of the 'install_config'
command, and by the boot modules specified at the 'build_boot_image' command.
Now that the integration is complete, the scenario is executed via the
'run_genode_until' command. This command takes a regular expression as
argument, which determines the successful termination of the test case. The
second argument is a timeout (is seconds). In the example, the test case will
fail if its output does not match the regular expression within the execution
time of 10 seconds.
The command 'append qemu_args' specifies run-script-specific qemu arguments in
the case that qemu is used to execute the scenario. This is the case for most
kernel platforms (except for Linux where core gets executed directly on the host).
Additional build-directory-specific qemu arguments can be specified in the
'etc/build.conf' file by defining the 'QEMU_OPT' variable. For example, to
prevent KVM being used on Ubuntu Linux, specify:
! QEMU_OPT = -no-kvm
To execute the run script from with build directory, you need to have Expect
installed. Typically, the Linux package is called 'expect'. Simply issue
the following command from within your build directory:
! make run/<run-script>
Note that you will need to have a GRUB 'stage2_eltorito' binary available
at '<genode-dir>/tool/grub' on base platforms that use an ISO image as boot
stategy.
Because the whole chain of actions, building, integrating, executing, and
validating an application scenario is now at the fingertips of issuing a
single command with no kernel-specific considerations needed, it has never
been easier to run the same scenario on a wide range of different kernels.
Please find further instructive examples at 'os/run/'. The 'ldso' run
script executes the test of the dynamic linker. It is completely generic.
The 'demo' run script starts Genode's default demo scenario and shows how
platform-specific considerations (e.g., which device drivers to use) can be
taken into account.
We found that the 'run' tool significantly boosted our productivity not
only for testing purposes but also for accelerating the development-test
cycle during our day-to-day work.
:Technical notes:
The 'run' tool uses Expect as automation tool. Expect is a Tcl interpreter,
which is accompanied by special functionality for automating interactive
command-line applications. Technically, a run script is an Expect script
which gets included by the 'tool/run' script. For the reference of
run-specific functions, please revise the documentation in the 'tool/run'
script. Because each run script is actual Expect source code, it is possible
to use all Tcl and Expect scripting features in a run script.
In particular, a run script may issue shell commands using Tcl's 'exec'
function. This way, even complex integration tasks can be accomplished.
For example, the integration of the Genode Live CD was done via a single
run script.
Build system
============
To facilitate the integration of 3rd-party build systems into the Genode build
process, we added support for pseudo targets that do not require any 'SRC'
declaration. Such 'target.mk' may contain custom rules that will be executed
when the target is revisited by the build system. The bindings are as follows:
! build_3rd_party:
! ...custom commands...
!
! $(TARGET): build_3rd_party
!
! clean_3rd_party:
! ...custom commands...
!
! clean_prg_objects: clean_3rd_party:

1289
doc/release_notes-11-05.txt
View File

File diff suppressed because it is too large Load Diff

703
doc/release_notes-11-08.txt
View File

@@ -1,703 +0,0 @@
===============================================
Release notes for the Genode OS Framework 11.08
===============================================
Genode Labs
One of Genode's most distinctive properties is its support for various
different kernels as base platforms. Each of the 8 currently supported kernels
differs with regard to features, security, hardware support, complexity, and
resource management. Even though different applications call for different
kernel properties, through Genode, those properties can be leveraged using a
unified API. The growing number of supported base platforms, however, poses two
challenges, which are the comprehension of the large diversity of tools and
boot concepts, and capturing of the semantic differences of all the kernels.
With the version 11.08, the framework mitigates the former challenge by
introducing a unified way to download, build, and use each of the
kernels with Genode's user-level infrastructure. The new tools empower users of
the framework to instantly change the underlying kernel without the need to know
the peculiarities of the respective kernels. Using microkernels has never been
easier.
The second challenge of translating each kernel's specific behaviour to the
framework's unified API longs for an automated testing infrastructure that
systematically exercises all the various facets of the API on all base
platforms. The new version introduces the tooling support especially designed
for conducting such quality-assurance measures. These tools largely remove the
burden of manual testing while helping us to uphold the stability and quality
of the framework as it grows in terms of functional complexity and number of
base platforms.
Speaking of functional enhancements, the work on version 11.08 was focused
on our block-device infrastructure and ARM support. The block-device-related
work is primarily motivated by our fundamental goal to scale Genode to a
general-purpose computing platform. The additions comprise new drivers for
SD-cards, IDE, SATA, USB storage as well as a new partition server. All those
components provide Genode's generic block interface, which is meant to be used
as back end for file systems. On file-system level, a new libc plugin utilizes
libffat to enable the straight-forward use of VFAT partitions by libc-using
programs.
The current release comes with far-reaching improvements with respect to
ARM-based platforms. The paravirtualized L4Linux kernel has been updated to
Linux version 2.6.39 running on both x86_32 and ARM. Also, Qt4 including Webkit
has become functional on ARMv6-based platforms.
Among the further improvements are many new examples in the form of
ready-to-use run scripts as well as a comprehensive documentation update.
Originally, we had planned to complement the Noux runtime environment to
support interactive command-line applications by the time of the current
release. However, we realized that the current users of the framework would
value the new streamlined tooling support, the enhanced documentation, and the
new quality-assurance infrastructure over such a functional addition. Hence, we
prioritized the topics accordingly. Even though you will find the first bits of
interactive GNU application support in this release, we deferred working on
this topic in full steam to the upcoming version 11.11.
Blurring the boundaries between different kernels
#################################################
Before the Genode project was born, each microkernel carried along its own
userland. For example, the L4/Fiasco kernel came with the L4 environment, the
OKL4 kernel came with Iguana, or the L4ka::Pistachio kernel came with a small
set of example components. Those user-level counterparts of the kernel
complemented their respective kernels with a runtime for user-level
applications and components while exposing significant parts of the kernel
interface at API level. Consequently, most if not all applications developed
against these APIs were tied to a particular kernel. On the one hand, this
approach enabled developers to fine-tune their programs using kernel-specific
features. On the other hand, much effort was wasted by duplicating other
people's work. Eventually, all of the mentioned userlands stayed limited to
special purposes - for the most part the purposes of operating-systems
researchers. Consequently, none of the microkernels gained much attention in
general-purpose computing. Another consequence of the highly fragmented
microkernel community was the lack of a common ground to compare different
kernels in an unbiased way because each userland provided a different set of
components and libraries.
Different application areas call for different kernel features such as
security mechanisms, scheduling, resource management, and hardware support.
Naturally, each kernel exhibits a specific profile of these parameters
depending on its primary purpose. If one microkernel attempted to accommodate
too many features, it would certainly sacrifice the fundamental idea of being
minimally complex. Consequently, kernels happen to be vastly different. During
the past three years, however, Genode has demonstrated that one carefully
crafted API can target highly diverse kernels, and thereby enables users of
the framework to select the kernel that fits best with the requirements
dictated by each application scenario individually. For us Genode developers,
it was extremely gratifying to see that kernels as different as Linux and NOVA
can be reconciled at the programming-interface level. Still, each kernel comes
with different tools, configuration mechanisms, and boot concepts. Even though
Genode programs can be developed in a kernel-independent way, the deployment of
such programs still required profound insights into the peculiarities of the
respective kernel.
With the current release, we introduce a fundamentally new way of using
different microkernels by unifying the procedures of downloading and building
kernels as well as integrating and running Genode programs with each of them.
Existing Genode application scenarios can be ported between kernels in an
instant without the need for deep insights into the kernel's technicalities. As
a teaser, consider the following commands for building and running Genode's
graphical demo scenario on the OKL4 microkernel:
! # check out Genode
! svn co https://genode.svn.sourceforge.net/svnroot/genode/trunk genode
!
! # download the kernel, e.g., OKL4
! make -C genode/base-okl4 prepare
!
! # create Genode build directory
! genode/tool/create_builddir \
! okl4_x86 BUILD_DIR=build
!
! # build everything and execute the interactive demo
! make -C build run/demo
The same principle steps can be used for any of the OKL4, NOVA,
L4/Fiasco, Fiasco.OC, L4ka::Pistachio, or Codezero kernels. You should
nevertheless consult the documentation at 'base-<platform>/doc/' before
starting to use a specific kernel because some base platforms require
the installation of additional tools.
Under the hood, this seamless way of dealing with different kernels is made
possible by the following considerations:
:Repository preparation:
Each kernel comes from a different source such as a Git/SVN/Mercurial
repository or a packaged archive. Some kernels require additional patches. For
example, OKL4 needs to be patched to overcome problems with modern tool chains.
Now, each 'base-<platform>' repository hosts a 'Makefile' that automates the
download and patch procedure. To download the source code of a kernel,
issue 'make prepare' from within the kernel's 'base-<platform>' directory. The
3rd-party source code will be located at 'base-<platform>/contrib/'.
:Building the kernel:
Each kernel has a different approach when it comes to configuration and
compilation. For example, NOVA comes with a simple 'Makefile', OKL4 relies on a
complex SCons-based build system, L4ka::Pistachio uses CML2 and autoconf (for
the userland tools). Furthermore, some kernels require the setting of specific
configuration values. We have streamlined all these procedures into the Genode
build process by the means of a 'kernel' pseudo target and a 'platform' pseudo
library. The kernel can be compiled directly from the Genode build directory by
issuing 'make kernel'. The 'platform' pseudo library takes care of making the
kernel headers available to Genode. For some kernels such as OKL4 and NOVA, we
replaced the original build mechanism with a Genode target. For other kernels
such as L4ka::Pistachio or Fiasco.OC, we invoke the kernel's build system.
:Genode build directory:
Genode build directories are created via the 'tool/create_builddir' tool.
This tool used to require certain kernel-specific arguments such as the
location of the kernel source tree. Thanks to the unified way of preparing
kernels, the need for such arguments has vanished. Now, the only remaining
arguments to 'create_builddir' are the actual platform and the location
of the build directory to create.
:System integration and booting:
As diverse the build systems of the kernels are, so are the boot concepts. Some
kernels rely on a multiboot-compliant boot loader whereas others have special
tools for creating boot images. Thankfully, Genode's run concept allows us to
hide the peculiarities of booting behind a neat and easy-to-use facade. For
each platform we have crafted a dedicated run environment located at
'base-<platform>/run/env', which contains the rules for system integration and
booting. Therefore, one and the same run script can be used to build and
execute one application scenario across various different kernels. For an
illustrative example, the 'os/src/run/demo.run' script can be executed on all
base platforms (except for base-mb) by issuing 'make run/demo' from within the
build directory.
Emerging block-device infrastructure
####################################
Since version 10.08, Genode is equipped with a block-session interface. Its
primary use cases so far were the supply of the paravirtualized OKLinux kernel
with backing store, and the access of the content of a bootable Live CD.
However, for our mission to use Genode as general-purpose computing platform,
disk device access is crucial. Therefore, we dedicated our attention to
various aspects of Genode's block-device infrastructure, reaching from
programming APIs for block drivers, over partition handling, to file-system
access.
:Block session interface:
The glue that holds all block-device-related components together is the generic
block interface 'os/include/block_session'. It is based on the framework's
packet-stream facility, which allows the communication of bulk data via shared
memory and a data-flow protocol using asynchronous notifications. The interface
supports arbitrary allocation schemes and the use of multiple outstanding
requests. Hence, it is generally suited for scatter-gather DMA and the use of
command queuing as offered by the firmware of modern block-device controllers.
(albeit the current drivers do not exploit this potential yet)
:Block component framework:
Our observation that components implementing the block session interface share
similar code patterns prompted us to design a framework API for implementing
this family of components. The set of classes located at 'os/include/block'
facilitate the separation of device-specific code from application logic.
Whereas 'component.h' provides the application logic needed to implement the
block service, the 'driver.h' is an abstract interface to be implemented by the
actual device driver. This new infrastructure significantly reduces code
duplication among new block-device drivers.
:Device-driver implementations:
The new block-device drivers introduced with the current release address
common types of block devices:
* By adding ATA read/write support to the ATAPI driver ('os/src/drivers/atapi'),
this driver can be used to access IDE disks now.
* The new fully-functional SD-card driver ('os/src/drivers/sdcard') enables the
use of SD-cards connected via the PL180 controller.
* The USB storage driver ('linux_drivers/src/drivers/usb') has been adapted
to the block-session interface and can be used on PC hardware.
* The new AHCI driver ('os/src/drivers/ahci') enables the access of disks
connected via SATA on PC hardware.
Because all drivers are providing the generic block-session interfaces, they
can be arbitrarily combined with components that use this interface as back
end, for example, the partition server and file systems.
:Partition manager as resource multiplexer:
The new partition manager ('os/src/server/part_blk') multiplexes one back-end
block session to multiple block sessions, each accessing a different partition.
Its natural role is being "plugged" between a block-device driver and a file
system.
:File-system access:
Even though a session interface for file systems does not exist yet, we
enabled the use of VFAT partitions through a libc plugin. This libc plugin uses
the ffat library to access files stored on a block device. An
application using this plugin can be directly connected to a block session.
New documentation
#################
The new way of dealing with different kernels motivated us to revisit and
complement our exiting documentation. The following documents are new or
have received considerable attention:
:[http://genode.org/documentation/developer-resources/getting_started - Getting started]:
The revised guide of how to explore Genode provides a quick way to
test drive Genode's graphical demo scenario with a kernel of your
choice and gives pointers to documents needed to proceed your
exploration.
:[http://genode.org/documentation/developer-resources/build_system - Build system manual]:
The new build-system manual explains the concepts behind Genode's
build system, provides guidance with creating custom programs and
libraries, and covers the tool support for the automated integration
and testing of application scenarios.
:[http://genode.org/documentation/components - Components overview]:
The new components-overview document explains the categorization of
Genode's components and lists all components that come with the framework.
:[http://genode.org/documentation/developer-resources/init - Configuration of the init process]:
The document describes Genode's configuration concept, the routing of
service requests, and the expression of mandatory access-control policies.
:[http://genode.org/community/wiki - Wiki]:
The platform-specific Wiki pages for L4/Fiasco, L4ka::Pistachio, NOVA,
Codezero, Fiasco.OC, and OKL4 have been updated to reflect the new flows of
working with the respective base platforms.
Base framework
##############
The RPC API for performing procedure calls across process boundaries
introduced with the version 11.05 was the most significant API change
in Genode's history. To make the transition from the old client-server
API to the new RPC API as smooth as possible, we temporarily upheld
compatibility to the old API. Now, the time has come to put the old
API at rest. The changes that are visible at API level are as follows:
* The old client-server API in the form of 'base/server.h' is no more.
The functionality of the original classes 'Server_entrypoint' and
'Server_activation' is contained in the 'Rpc_entrypoint' class provided
via 'base/rpc_server.h'.
* When introducing the RPC API, we intentionally left the actual session
interfaces as unmodified as possible to proof the versatility of the new
facility. However, it became apparent that some of the original interfaces
could profit from using a less C-ish style. For example, some interfaces used
to pass null-terminated strings as 'char const *' rather than via a dedicated
type. The methodology of using the new RPC API while leaving the original
interfaces intact was to implement such old-style functions as wrappers
around new-style RPC functions. These wrappers were contained in
'rpc_object.h' files, e.g. for 'linux_dataspace', 'parent', 'root',
'signal_session', 'cpu_session'. Now, we have taken the chance to modernise
the API by disposing said wrappers. Thereby, the need for 'rpc_object.h'
files has (almost) vanished.
* The remaining users of the old client-server API have been adapted to the
new RPC API, most prominently, the packet-stream-related interfaces such as
'block_session', 'nic_session', and 'audio_session'.
* We removed 'Typed_capability' and the second argument of the 'Capability'
template. The latter was an artifact that was only used to support the
transition from the old to the new API.
* The 'ipc_client' has no longer an 'operator int'. The result of an IPC can
be requested via the 'result' function.
* We refined the accessors of 'Rpc_in_buffer' in 'base/rpc_args.h'. The
'addr()' has been renamed to 'base()', 'is_valid_string()' considers the
buffer's capacity, and the new 'string()' function is guaranteed to return a
null-terminated string.
* We introduced a new 'Rm_session::Local_addr' class, which serves two
purposes. It allows the transfer of the bit representation of pointers across
RPC calls and effectively removes the need for casting the return type of
'Rm_session::attach' to the type needed at the caller side.
* The 'Connection' class template has been simplified, taking the session
interface as template argument (rather than the capability type). This change
simplified the 'Connection' classes of most session interfaces.
* The never-used return value of 'Parent::announce' has been removed. From the
child's perspective, an announcement always succeeds. The way of how the
announcement is treated is entirely up to the parent. The client should never
act differently depending on the parent's policy anyway.
* The new 'Thread_base::cap()' accessor function allows obtaining the thread's
capability as used for the argument to CPU-session operations.
Operating-system services and libraries
#######################################
Dynamic linker
==============
As a follow-up to the major revision of the dynamic linker that was featured
with the previous release, we addressed several corner cases related to
exception handling and improved the handling of global symbols.
The dynamic linker used to resolve requests for global symbols by handing out
its own symbols if present. However, in some cases, this behaviour is
undesired. For example, the dynamic linker contains a small set of libc
emulation functions specifically for the ported linker code. In the presence of
the real libc, however, these symbols should never be considered at all. To
avoid such ambiguities during symbol resolution, the set of symbols to be
exported is now explicitly declared by the white-list contained in the
'os/src/lib/ldso/symbol.map' file.
We changed the linkage of the C++ support library ('cxx') against dynamic
binaries to be consistent with the other base libraries. Originally, the 'cxx'
library was linked to both the dynamic linker and the dynamic binary, which
resulted in subtle problems caused by the duplication of cxx-internal data
structures. By linking 'cxx' only to the dynamic linker and exporting the
'__cxa' ABI as global symbols, these issues have been resolved. As a positive
side effect, this change reduces the size of dynamic binaries.
C++ exception handling in the presence of shared libraries turned out to be
more challenging than we originally anticipated. For example, the
'_Unwind_Resume' symbol is exported by the compiler's 'libsupc++' as a hidden
global symbol, which can only be resolved when linking this library to the
binary but is not seen by the dynamic linker. This was the actual reason of why
we used to link 'cxx' against both dynamic binaries and shared libraries
causing the problem mentioned in the previous paragraph. Normally, this problem
is addressed by a shared library called 'libgcc_s.so' that comes with the
compiler. However, this library depends on glibc, which prevents us from using
it on Genode. Our solution is renaming the hidden global symbol using a
'_cxx__' prefix and introducing a non-hidden global wrapper function
('__cxx__Unwind_Resume' in 'unwind.cc'), which is resolved at runtime by the
dynamic linker.
Another corner case we identified is throwing exceptions from within the
dynamic linker. In contrast to the original FreeBSD version of the dynamic
linker, which is a plain C program that can never throw a C++ exception,
Genode's version relies on C++ code that makes use of exceptions. To support
C++ exceptions from within the dynamic linker, we have to relocate the
linkers's global symbols again after having loaded the dynamic binary. This
way, type information that is also present within the dynamic binary becomes
relocated to the correct positions.
Block partition server
======================
The new block-partition server uses Genode's block-session interfaces as both
front and back end, leading to the most common use case where this server will
reside between a block driver and a higher level component like a file-system
server.
At startup, the partition server will try to parse the master boot record (MBR)
of its back-end block session. If no partition table is found, the whole block
device is exported as partition '0'. In the other case, the MBR and possible
extended boot records (EBRs) are parsed and offered as separate block sessions
to the front-end clients. The four primary partitions will receive partition
numbers '1' to '4' whereas the first logical partition will be assigned to '5'.
The policy of which partition is exposed to which client can be expressed
in the config supplied to the 'part_blk' server. Please refer to the
documentation at 'os/src/server/part_blk/README' for further details. As an
illustration of the practical use of the 'part_blk' server, you can find a run
script at 'os/run/part_blk.run'.
Skeleton of text terminal
=========================
As part of the ongoing work towards using interactive text-based GNU software
on Genode, we created the first bits of the infrastructure required for
pursuing this quest:
The new terminal-session interface at 'os/include/terminal_session/' is the
designated interface to be implemented by terminal programs.
After investigating the pros and cons of various terminal protocols and
terminal emulators, we settled for implementing a custom terminal emulator
implementing the Linux termcap. This termcap offers a reasonable small set of
commands while providing all essential features such as function-key support
and mouse support. Thanks to Peter Persson for pointing us to the right
direction! The preliminary code for parsing the escape sequences for the Linux
termcap is located at 'gems/include/terminal/'.
We have created a simplistic terminal service that implements the
terminal-session interface using a built-in font. Please note that the
implementation at 'gems/src/server/terminal/' is at an early stage. It is
accompanied by a simple echo program located at 'gems/src/test/terminal_echo'.
Device drivers
##############
USB HID and USB storage
=======================
We replaced the former DDE-Linux-based USB-related driver libraries (at the
'linux_drivers/' repository) by a single USB driver server that offers the
'Input' and 'Block' services. This enables us to use both USB HID and USB
storage at the same time. The new USB driver is located at
'linux_drivers/src/drivers/usb/'.
For using the USB driver as input service (supporting USB HID), add the
'<hid/>' tag to the 'usb_drv' configuration. Analogously, for using the driver
as block service, add the '<storage/>' tag. Both tags can be combined.
For testing the USB stack, the 'linux_drivers' repository comes with the run
scripts 'usb_hid.run' and 'usb_storage.run'.
ATA read/write support
======================
The ATAPI driver has been extended to support IDE block devices for both
read and write transactions. To use the new facility, supply 'ata="yes"'
as XML attribute to the config node of 'atapi_drv'. Please note that this
driver was primarily tested on Qemu. Use it with caution.
SATA driver
===========
The new SATA driver at 'os/src/drivers/ahci/' implements the block-driver
API ('os/include/block'), thus exposing the block-session interface as
front-end. AHCI depends on Genode's PCI driver as well as the timer server. For
a usage example see: 'os/run/ahci.run'.
Limitations and known issues
----------------------------
Currently, the server scans the PCI bus at startup and retrieves the first available
AHCI controller, scans the controller ports and uses the first non-ATAPI port
where a device is present.
On real hardware and on kernels taking advantage of I/O APICs (namely NOVA and
Fiasco.OC) we still lack support for ACPI parsing and thus for interrupts,
leading to a non-working driver.
SD-card driver
==============
The first fragments of our SD-card driver that we introduced with the previous
release have been complemented. The new SD-card driver located at
'os/src/drivers/sd_card/' implements the block-session interface by using
MMC/SD-cards and the PL180 controller as back end. Currently the driver
supports single-capacity SD cards. Therefore, the block file for Qemu should
not exceed 512 MB. Because the driver provides the generic block-session
interface, it can be combined with the new 'libc_ffat' plugin in a
straight-forward way. To give the driver a quick spin, you may give the
'libports/run/libc_ffat.run' script on the 'foc_pbxa9' platform a try.
ARM Realview PL011 UART driver
==============================
The new PL011 UART driver at 'os/src/drivers/uart/' implements the LOG session
interface using the PL011 device. Up to 4 UARTs are supported. The assignment
of UARTs to clients can be defined via a policy supplied to the driver's config
node. For further information, please refer to the README file within the
'uart' directory.
Libraries and applications
##########################
Hello tutorial
==============
The 'hello_tutorial/' repository contains a step-by-step guide for building
a simple client-server scenario. The tutorial has been rewritten for the new
RPC API and is now complemented by a run script for testing the final scenario
on various base platforms.
C and C++ runtimes
==================
:Support for standard C++ headers:
Triggered by public demand for using standard C++ headers for Genode applications,
we introduced a generally usable solution in the form of the 'stdcxx' library
to the 'libc' repository. The new 'stdcxx' library is not a real library. (you
will find the corresponding 'lib/mk/stdcxx.mk' file empty) However, it comes
with a 'lib/import/import-stdcxx.mk' file that adds the compiler's C++ includes
to the default include-search path for any target that has 'stdcxx' listed in
its 'LIBS' declaration.
:Libc back end for accessing VFAT partitions:
The new 'libc_ffat' libc plugin uses a block session via the ffat library. It
can be used by a Genode application to access a VFAT file system via the libc
file API. The file-system access is performed via the 'ffat' library. To
download this library and integrate it with Genode, change to the 'libports'
repository and issue the following command:
! make prepare PKG=ffat
For an example of how to use the libc-ffat plugin, please refer to the run
script 'libports/run/libc_ffat.run'. The source code of the test program can be
found at 'libports/src/test/libc_ffat/'.
Qt4
===
Qt4 version 4.7.1 has been enabled on ARMv6-based platforms, i.e., PBX-A9 on
Fiasco.OC. The support comprises the entire Qt4 framework including qt_webcore
(Webkit).
L4Linux
=======
L4Linux enables the use of one or multiple instances of Linux-based operating
systems as subsystems running on the Fiasco.OC kernel. The Genode version of
L4Linux has seen the following improvements:
:Kernel version: has been updated to Linux 2.6.39.
:ARM support: The L4Linux kernel can be used on ARM-based platforms now.
The PBX-A9 platform is supported via the 'l4linux.run' script as found
at 'ports-foc/run/'. Please find more information at 'ports-foc/README'.
:Genode-specific stub drivers outside the kernel tree:
The stub drivers that enable the use of Genode's services as virtual
devices for L4Linux have been moved outside the kernel patch, which
makes them much easier to maintain. These stub drivers are located
under 'ports-foc/src/drivers/'.
Platform support
################
All base platforms are now handled in a unified fashion. Downloading 3rd-party
source code is performed using the 'prepare' rule of the 'Makefile' provided by
the respective kernel's 'base-<platform>' repository. Once, the platform's base
repository is prepared, the kernel can be built directly from the Genode
build directory using 'make kernel'. All base platforms are now supported by
Genode's run mechanism that automates the tasks of system integration and
testing. For more details about each specific kernel, please revisit the
updated documentation within the respective 'base-<platform>/doc/' directory.
:L4/Fiasco:
The kernel has been updated to revision 472, enabling the use of recent
GNU tool chains.
:Fiasco.OC:
The kernel as been updated to revision 36, which remedies stability problems
related to interaction of the IPC path with thread destruction. The new version
improves the stability of highly dynamic workloads that involve the frequent
creation and destruction of subsystems. However, we experienced the new kernel
version to behave instable on the x86_64 architecture. If you depend on x86_64,
we recommend to temporarily stick with Genode 11.05 and Fiasco.OC revision 31.
:L4ka::Pistachio:
The kernel has been updated to revision 803, enabling the use of recent
versions of binutils.
:OKL4:
OKL4v2 is showing its age. Apparently, the use of the original distribution
requires tools (i.e., python 2.4) that do not ship with current Linux
distributions anymore. This makes it increasingly difficult to use this kernel.
Still, we find ourselves frequently using it for our day-to-day development. To
streamline the use of OKL4v2, we have now incorporated the kernel compilation
into the Genode build system and thereby weakened the kernel's dependency on
ancient tools. However, we decided to drop support for OKL4/ARM for now. We
figured that the supported GTA01 platform is hardly used anymore and hard to
test because it is unsupported by Qemu. Newer ARM platforms are supported by
other kernels anyway.
:Codezero:
Even though B-Labs apparently abandoned the idea of developing the Codezero
kernel in the open, we adapted Genode to the kernel's most recent Open-Source
version that is still available at the official Git repository. Furthermore,
the kernel is now fully supported by Genode's new 'make prepare' procedure and
run environment. Therefore, run scripts such as 'run/demo' can now easily be
executed on Codezero without the need to manually configure the kernel.
Note that, for now, we have disabled Codezero's capabilities because they do
not allow the assignment of device resources. Consequently, 'sys_map' fails for
MMIO regions when performing the capability check (calling 'cap_map_check').
Furthermore, the current version of the kernel requires a workaround for a
current limitation regarding the definition of a thread's pager. At some point,
Codezero abandoned the facility to define the pager for a given thread via the
exregs system call. Instead, the kernel hard-wires the creator of the thread as
the thread's pager. This is conflicting with Genode's way of creating and
paging threads. In the current version of Genode for this kernel, all threads
are paged by one thread (thread 3 happens to be the global pager) within core.
As a workaround to Codezero's current limitation, we define thread 3 to be the
pager of all threads. The patch of the upstream code is automatically being
applied by the 'make prepare' mechanism.
Build system and tools
######################
In addition to the major change with respect to the integration of the various
base platforms, Genode's tool support received the following incremental
improvements:
Build system
============
:Simplification of 'create_builddir' tool:
The 'create_builddir' tool has been relocated from
'tool/builddir/create_builddir' to 'tool/create_builddir' to make it more
readily accessible. Furthermore, we simplified the usage of the tool by
removing the mandatory 'GENODE_DIR' argument. If not explicitly specified, the
tool deduces 'GENODE_DIR' from the its known location within the Genode source
tree.
:Booting from USB sticks:
For most x86-based base platforms, their respective run environments execute
Genode from an ISO image via Qemu. Naturally, such an ISO image can be burned
onto a CD-ROM to be used to boot a real machine. However, booting from CD-ROM
is slow and optical drives are becoming scarce. Therefore we changed the
procedure of creating ISO images to support writing the resulting images to a
USB stick. Under the hood, the boot mechanism chain-loads GRUB via ISOLinux.
The files to implement the boot concept are located at 'tool/boot/'.
:Support for source files in target sub directories:
Until now, the 'SRC_*' declarations in target description files contained
a list of plain file names. The location of the files within the directory
tree had to be defined via 'vpath'. This led to inconveniences when building
3rd-party code that contains files with the same name at different subdirectories.
To resolve such an ambiguity, the target had to be decomposed into multiple
libraries each building a different set of subdirectories. To make the
build system more convenient to use, we have now added support for specifying
source codes with a relative pathname. For example, instead of using
! SRC_CC = main.cc addon.cc
! vpath addon.cc $(PRG_DIR)/contrib
we can now use
! SRC_CC = main.cc contrib/addon.cc
Automated testing across multiple kernels
=========================================
To execute one or multiple test cases on more than one base platform, we
introduced a dedicated tool located at 'tool/autopilot'. Its primary purpose is
the nightly execution of test cases. The tool takes a list of platforms and a
list of run scripts as arguments and executes each run script on each platform.
The build directory for each platform is created at
'/tmp/autopilot.<username>/<platform>' and the output of each run script is
written to a file called '<platform>.<run-script>.log'. On stderr, autopilot
prints the statistics about whether or not each run script executed
successfully on each platform. If at least one run script failed, autopilot
returns a non-zero exit code, which makes it straight forward to include
autopilot into an automated build-and-test environment.

1008
doc/release_notes-11-11.txt
View File

File diff suppressed because it is too large Load Diff

862
doc/release_notes-12-02.txt
View File

@@ -1,862 +0,0 @@
===============================================
Release notes for the Genode OS Framework 12.02
===============================================
Genode Labs
The release of Genode 12.02 marks an exciting point in the history of the
project as it is the first version developed in the open rather than within the
chambers of Genode Labs. Thereby, we have embraced GitHub as central facility
for discussion and source-code management. This change has benefits for users
and developers of the framework alike. For users, it has become possible to get
hold of the latest developments using the official 'genodelabs/master' branch and
get involved with discussing the current activities. For regular Genode
developers, the public Git repository replaces a former mix of public
Subversion and company-internal Mercurial repositories, making life much
easier. In Section [Liberation of the development process], we outline the
motivation behind this change and give pointers to the new resources.
The major new additions to the base system are a new framework API for accessing
memory-mapped I/O resources, special support for using Genode as user-level
component framework on Linux, and API support for the reuse of existing
components in the form of sandboxed libraries. These changes are accompanied
with new device-driver infrastructure such as the first version of a device
driver manager and a new ACPI parser.
Feature-wise, the current release takes the first steps towards the goal of the
[http://genode.org/about/road-map - Roadmap for 2012], turning Genode into a
general-purpose OS ready for everyday use by its developers. According to the
roadmap, we enhanced the Noux runtime with fork semantics so that we can run
command-line based GNU programs such as the bash shell and coreutils unmodified
and natively on various microkernels. Furthermore, the library infrastructure
has been enhanced by porting and updating libraries such as Qt 4.7.4 and the
MuPDF PDF rendering engine.
Liberation of the development process
#####################################
In summer 2011, we started a discussion within Genode Labs about changing the
mode of how Genode is developed. Until then, most design discussions and the
actual development work took place locally at the company. At quarterly
intervals, we used to publish our work in the form of official Genode
releases. This way of development seemed to work quite well for us, we were
satisfied about the pace of development, and with each release, our project got
more recognition.
However, the excellent book [http://producingoss.com/ - Producing Open Source Software]
made us realize that even though we released our work under an Open-Source
license, our development process was actually far from being open and may have
discouraged participation of people outside the inner circle of developers.
Because we believe that the framework has reached a state where it becomes
interesting for a wider audience of users and developers, the idea was born
to liberate the project from its closed fashion of development.
In the beginning of December, the vague idea has become a plan. So we finally
brought the topic to our mailing list
([http://genode.org/news/steps-towards-an-open-development-process - Steps towards an open development process]).
We decided to take the release cycle for Genode 12.02 as the opportunity to put
our plan to practice. The central element of this endeavour was moving the
project over to GitHub and adapt our workflows and tooling support accordingly.
First, we started to embrace GitHub's issue tracker for the management of
working topics:
:[http://github.com/genodelabs/genode/issues]: Issue Tracker
The most significant step was leaving our Genode-Labs-internal code
repositories behind and starting a completely public Git repository instead:
:[https://github.com/genodelabs]: Genode Labs at GitHub
With the code repository going public, the way was cleared to opening up design
discussions as well. Instead of having such discussions internally at Genode
Labs, we try to increasingly take them to our mailing list and issue tracker.
With this new way of development, we hope to make the project much more
approachable for people who want to get involved and let Genode reach far out
beyond the reach of our little company.
The changes mentioned above are actually just the tip of the iceberg. For
example, the transition phase required us to rethink the way the project
website is maintained. From now on, almost all of the content of genode.org
comes directly from the project's Git repository. So maintaining website
content is done in the same coherent and transparent way as working on Genode's
code base. So we could finally put the old Wiki to rest. In the process, we
largely revisited the existing content. For example, we rewrote the
[http://genode.org/community/contributions - contributions] document in a
tutorial-like style and incorporated several practical hints, in particular
related to the recommended use of Git.
Although it is probably too early to judge the outcome of our transition, we
are excited how smooth this massive change went. We attribute this pleasant
experience mostly to the excellent GitHub hosting platform, which instantly
ignited a spirit of open collaboration among us. We are excited to see new
people approaching us and showing their interest for teaming up, and we are
curious about where this new model of development will take Genode in the
future.
Base framework, low-level OS infrastructure
###########################################
RPC framework refinements
=========================
Until now, the RPC framework did not support const RPC functions. Rather than
being a limitation inherent to the concept, const RPC functions plainly did not
exist. So supporting them was not deemed too important. However, there are uses
of RPC interfaces that would benefit from a way to declare an RPC function as
const. Candidates are functions like 'Framebuffer::Session::mode()' and
'Input::Session::is_pending()'.
With the current version, we clear the way towards declaring such functions as
const. Even though the change is pretty straight-forward, the thorough support
for const-qualified RPC functions would double the number of overloads for the
'call_member' function template (in 'base/include/util/meta.h'). For this
reason, as of now, the support of const functions is limited to typical getter
functions with no arguments. This appears to be the most common use of such
functions.
API support for enslaving services
==================================
While evolving and using the framework, we always keep an eye on recurring
patterns of how its API is used. Once such a pattern becomes obvious, we try
to take a step back, generalize the observed pattern, and come up with a new
building block that unifies the former repetitive code fragments.
One of those patterns that was far from obvious when we designed Genode years
ago is the use of a service running as child of its own client. At the first
glance, this idea seems counter-intuitive because normally, services are
understood as components that operate independently and protected from their
(untrusted) clients. But there is a class of problems where this approach
becomes extremely useful: The reuse of protocol implementations as a
library-like building block. Most services are actually protocol stacks that
translate a low-level protocol to a more abstract API. For example, a block
device driver translates a specific device API to the generic 'Block_session'
interface. Or the 'iso9660' service translates the 'Block_session' interface to
the 'Rom_session' interface by parsing the ISO9660 file system. Or similarly,
the 'tar_rom' service parses the tar file format to make its content available
via the 'Rom_session' interface.
If a particular functionality is needed by multiple programs, it is common
practice to move this functionality into a dedicated library to avoid the
duplication of the same code at many places. For example, if a program would
need to parse a tar archive, it might be tempting to move the tar-parsing code
from the 'tar_rom' service into a dedicated library, which can then be used by
both the 'tar_rom' service and the new program. An alternative approach is to
just re-use the 'tar_rom' service as a black box and treat it like it was a
library. That is, start the 'tar_rom' service as a child process, supply the
resources the component needs to operate and, in turn, use its API (now in the
form of an RPC interface) to get work done. Because the service is started as a
mere tool at the discretion of its client, we call it *slave*. It turns out
that this idea works exceedingly well in many cases. In a way, it resembles the
Unix philosophy to solve complex problems by combining many small tools each
with a specific purpose. In contrast to the use of libraries, the reuse of
entire components has benefits with regard to fault isolation. Because the
reused functionality is sandboxed within a distinct process, the environment
exposed to this code can be tailored to a rigid subset of the host program's
environment. In the event of a fault within the reused component, the reach of
problem is therefore limited.
On the other hand, we observed that even though this idea works as intended,
implementing the idea for a particular use case involved a good deal of
boiler-plate code where most of this code is needed to define the slave's
environment and resources. Hence, we reviewed the existing occurrences of the
slave pattern and condensed their common concerns into the 'Slave_policy' and
'Slave' classes residing in 'os/include/os/slave.h'. The 'Slave' class is meant
to be used as is. It is merely a convenience wrapper for a child process and
its basic resources. The 'Slave_policy' is meant as a hook for service-specific
customizations. The best showcase is the new 'd3m' component located at
'gems/src/server/d3m'. D3m extensively uses the slave pattern by instantiating
and destroying drivers and file-system instances on-the-fly. A further instance
of this pattern can be found in the new ACPI driver introduced with the current
release.
Support for resizable framebuffers
==================================
The framebuffer-session interface has remained largely untouched since the
original release of Genode in 2008. Back then, we were used to rely on C-style
out parameters in RPC functions. The current RPC framework, however, promotes
the use of a more object-oriented style. So the time has come to revisit the
framebuffer session interface. Instead of using C-style out parameters, the new
'mode()' RPC call returns the mode information as an object of type 'Mode'.
Consequently, mode-specific functions such as 'bytes_per_pixel()' have been
moved to the new 'Framebuffer::Mode' class. The former 'info()' function is
gone.
In addition to the overhaul of the RPC interface, we introduced basic support
for resizable framebuffers. The new 'mode_sigh()' function enables a client to
register a signal handler at the framebuffer session. This signal handler gets
notified in the event of server-side mode changes. Via the new 'release()'
function, the client is able to acknowledge a mode change. By calling it, the
client tells the framebuffer service that it no longer uses the original
framebuffer dataspace. So the server can replace it by a new one. After having
called 'release()', the client can obtain the dataspace for the new mode by
calling 'dataspace()' again.
MMIO access framework
=====================
As the arsenal of native device drivers for Genode grows, we are observing
an increased demand to formalize the style of how drivers are written to
foster code consistency. One particular cause of inconsistency used to be
the way of how memory-mapped I/O registers are accessed. C++ has poor support
for defining bit-accurate register layouts in memory. Therefore, driver code
usually carries along a custom set of convenience functions for reading and
writing registers of different widths as well as a list of bit definitions in
the form of enum values or '#define' statements. To access parts of a register,
the usual pattern is similar to the following example (taken from the pl011
UART driver:
! enum {
! UARTCR = 0x030, /* control register */
! UARTCR_UARTEN = 0x0001, /* enable bit in control register */
! ...
! }
! ...
!
! /* enable UART */
! _write_reg(UARTCR, _read_reg(UARTCR) | UARTCR_UARTEN);
This example showcases two inconveniences: The way the register layout is
expressed and the manual labour needed to access parts of registers. In the
general case, a driver needs to also consider 'MASK' and 'SHIFT' values to
implement access to partial registers properly. This is not just inconvenient
but also error prone. For lazy programmers as ourselves, it's just too easy to
overwrite "undefined" bits in a register instead of explicitly masking the
access to the actually targeted bits. Consequently, the driver may work fine
with the current generation of devices but break with the next generation.
So the idea was born to introduce an easy-to-use formalism for this problem. We
had two goals: First, we wanted to cleanly separate the declaration of register
layouts from the program logic of the driver. The actual driver program should
be free from any intrinsics in the form of bit-masking operations. Second, we
wanted to promote uncluttered driver code that uses names (i.e., in the form of
type names) rather than values to express its operations. The latter goal is
actually achieved by the example above by the use of enum values, but this is
only achieved through discipline. We would prefer to have an API that
facilitates the use of proper names as the most convenient way to express an
operation.
The resulting MMIO API comes in the form of two new header files located at
'base/include/util/register.h' and 'base/include/util/mmio.h'.
Register declarations
~~~~~~~~~~~~~~~~~~~~~
The class templates found in 'util/register.h' provide a means to express
register layouts using C++ types. In a way, these templates make up for
C++'s missing facility to define accurate bitfields. Let's take a look at
a simple example of the 'Register' class template that can be used to define
a register as well as a bitfield within this register:
! struct Vaporizer : Register<16>
! {
! struct Enable : Bitfield<2,1> { };
! struct State : Bitfield<3,3> {
! enum{ SOLID = 1, LIQUID = 2, GASSY = 3 };
! };
!
! static void write (access_t value);
! static access_t read ();
! };
In the example, 'Vaporizer' is a 16-bit register, which is expressed via the
'Register' template argument. The 'Register' class template allows for
accessing register content at a finer granularity than the whole register
width. To give a specific part of the register a name, the 'Register::Bitfield'
class template is used. It describes a bit region within the range of the
compound register. The bit 2 corresponds to true if the device is enabled and
bits 3 to 5 encode the 'State'. To access the actual register, the methods
'read()' and 'write()' must be provided as backend, which performs the access
of the whole register. Once defined, the 'Vaporizer' offers a handy way to
access the individual parts of the register, for example:
! /* read the whole register content */
! Vaporizer::access_t r = Vaporizer::read();
!
! /* clear a bit field */
! Vaporizer::Enable::clear(r);
!
! /* read a bit field value */
! unsigned old_state = Vaporizer::State::get(r);
!
! /* assign new bit field value */
! Vaporizer::State::set(r, Vaporizer::State::LIQUID);
!
! /* write whole register */
! Vaporizer::write(r);
Memory-mapped I/O
~~~~~~~~~~~~~~~~~
The utilities provided by 'util/mmio.h' use the 'Register' template class as
a building block to provide easy-to-use access to memory-mapped I/O registers.
The 'Mmio' class represents a memory-mapped I/O region taking its local base
address as constructor argument. Let's take a simple example to see how it is
supposed to be used:
! class Timer : Mmio
! {
! struct Value : Register<0x0, 32> { };
! struct Control : Register<0x4, 8> {
! struct Enable : Bitfield<0,1> { };
! struct Irq : Bitfield<3,1> { };
! struct Method : Bitfield<1,2>
! {
! enum { ONCE = 1, RELOAD = 2, CYCLE = 3 };
! };
! };
!
! public:
!
! Timer(addr_t base) : Mmio(base) { }
!
! void enable();
! void set_timeout(Value::access_t duration);
! bool irq_raised();
! };
The memory-mapped timer device consists of two registers: The 32-bit 'Value'
register and the 8-bit 'Control' register. They are located at the MMIO offsets
0x0 and 0x4, respectively. Some parts of the 'Control' register have specific
meanings as expressed by the 'Bitfield' definitions within the 'Control'
struct.
Using these declarations, accessing the individual bits becomes almost a
verbatim description of how the device is used. For example:
! void enable()
! {
! /* access an individual bitfield */
! write<Control::Enable>(true);
! }
!
! void set_timeout(Value::access_t duration)
! {
! /* write complete content of a register */
! write<Value>(duration);
!
! /* write all bitfields as one transaction */
! write<Control>(Control::Enable::bits(1) |
! Control::Method::bits(Control::Method::ONCE) |
! Control::Irq::bits(0));
! }
!
! bool irq_raised()
! {
! return read<Control::Irq>();
! }
In addition to those basic facilities, further noteworthy features of the new
API are the support for register arrays and the graceful overflow handling
with respect to register and bitfield boundaries.
C Runtime
=========
We extended our FreeBSD-based C runtime to accommodate the needs of the Noux
runtime environment and our port of the MuPDF application.
* The dummy implementation of '_ioctl()' has been removed. This function is
called internally within the libc, i.e., by 'tcgetattr()'. For running
libreadline in Noux, we need to hook into those ioctl operations via the
libc plugin interface.
* The 'libc/regex' and 'libc/compat' modules have been added to the libc.
These libraries are needed by the forthcoming port of Slashem to Noux.
* We added a default implementation of 'chdir()'. It relies on the sequence of
'open()', 'fchdir()', 'close()'.
* The new libc plugin 'libc_rom' enables the use of libc file I/O functions
to access ROM files as provided by Genode ROM session.
* We changed the libc dummy implementations to always return ENOSYS. Prior
this change, 'errno' used to remain untouched by those functions causing
indeterministic behaviour of code that calls those functions, observes the
error return value (as returned by most dummies), and evaluates the error
condition reported by errno.
* If using the libc for Noux programs, the default implementations of
time-related functions such as 'gettimeofday()' cannot be used because they
rely on a dedicated timeout-scheduler thread. Noux programs, however, are
expected to contain only the main thread. By turning the functions into weak
symbols, we enabled the noux libc-plugin to provide custom implementations.
DDE Kit
=======
Linux DDE used to implement Linux spin locks based on 'dde_kit_lock'. This
works fine if a spin lock is initialized only once and used from then on. But
if spin locks are initialized on-the-fly at a high rate, each initialization
causes the allocation of a new 'dde_kit_lock'. Because in contrast to normal
locks, spinlocks cannot be explicitly destroyed, the spin-lock emulating locks
are never freed. To solve the leakage of locks, we complemented DDE Kit with
the new 'os/include/dde_kit/spin_lock.h' API providing the semantics as
expected by Linux drivers.
Libraries and applications
##########################
New and updated libraries
=========================
:Qt4 updated to version 4.7.4:
We updated Qt4 from version 4.7.1 to version 4.7.4. For the most part, the
update contains bug fixes as detailed in the release notes for the versions
[http://qt.nokia.com/products/changes/changes-4.7.2 - 4.7.2],
[http://qt.nokia.com/products/changes/changes-4.7.3 - 4.7.3], and
[http://labs.qt.nokia.com/2011/09/01/qt-4-7-4-released - 4.7.4].
:Update of zlib to version 1.2.6:
Because zlib 1.2.5 is no more available at zlib.net, we bumped the zlib
version to 1.2.6.
:New ports of openjpeg, jbig2dec, and mupdf:
MuPDF is a fast and versatile PDF rendering library with only a few
dependencies. It depends on openjpeg (JPEG2000 codec) and jbig2dec (b/w image
compression library). With the current version, we integrated those libraries
alongside the MuPDF library to the 'libports' repository.
GDB monitor refinements and automated test
==========================================
We improved the support for GDB-based user-level debugging as introduced with
the previous release.
For the x86 architecture, we fixed a corner-case problem with using the
two-byte 'INT 0' instruction for breakpoints. The fix changes the breakpoint
instruction to the single-byte 'HLT'. 'HLT' is a privileged instruction and
triggers an exception when executed in user mode.
The new 'gdb_monitor_interactive.run' script extends the original
'gdb_monitor.run' script with a startup sequence that automates the
initial break-in at the 'main()' function of a dynamically linked binary.
The revised 'gdb_monitor.run' script has been improved to become a full
automated test case for GDB functionalities. It exercises the following
features (currently on Fiasco.OC only):
* Breakpoint in 'main()'
* Breakpoint in a shared-library function
* Stack trace when not in a syscall
* Thread info
* Single stepping
* Handling of segmentation-fault exception
* Stack trace when in a syscall
PDF viewer
==========
According to our road map for 2012, we pursued the port of an existing PDF
viewer as native application to Genode.
We first looked at the [http://poppler.freedesktop.org - libpoppler],
which seems to be the most popular PDF rendering engine in the world of
freedesktop.org. To get a grasp on what the porting effort of this engine may
be, we looked at projects using this library as well as the library source
code. By examining applications such as the light-weight epdfview
application, we observed that libpoppler's primary design goal is to integrate
well with existing freedesktop.org infrastructure rather than to reimplement
functionality that is provided by another library. For example, fontconfig is
used to obtain font information and Cairo is used as rendering backend. In the
context of freedesktop.org, this makes perfect sense. But in our context,
porting libpoppler would require us to port all this infrastructure to Genode
as well. To illustrate the order of magnitude of the effort needed, epdfview
depends on 65 shared libraries. Of course, at some point in the future, we will
be faced to carry out this porting work. But for the immediate goal to have a
PDF rendering engine available on Genode, it seems overly involved. Another
criterion to evaluate the feasibility of integrating libpoppler with Genode is
its API. By glancing at the API, it seems to be extremely feature rich and
complex - certainly not a thing to conquer in one evening with a glass of wine.
The Qt4 backend of the library comprises circa 8000 lines of code. This value
can be taken as a vague hint at the amount of work needed to create a custom
backend, i.e., for Genode's framebuffer-session interface.
Fortunately for us, there exists an alternative PDF rendering engine named
MuPDF. The concept of MuPDF is quite the opposite of that of libpoppler.
MuPDF tries to be as self-sufficient as possible in order to be suitable
for embedded systems without comprehensive OS infrastructure. It comes with a
custom vector-graphics library (instead of using an existing library such as
Cairo) and it even has statically linked-in all font information needed to
display PDF files that come with no embedded fonts. That said, it does not
try to reinvent the wheel in every regard. For example, it relies on
common libraries such as zlib, libpng, jpeg, freetype, and openjpeg. Most
of them are already available on Genode. And the remaining libraries are rather
free-standing and easy to port. To illustrate the low degree of dependencies,
the MuPDF application on GNU/Linux depends on only 15 shared libraries. The
best thing about MuPDF from our perspective however, is its lean and clean API,
and the wonderfully simple example application. Thanks to this example, it was
a breeze to integrate the MuPDF engine with Genode's native framebuffer-session
and input-session interfaces. The effort needed for this integration work lies
in the order of less than 300 lines of code.
At the current stage, the MuPDF rendering engine successfully runs on Genode
in the form of a simple interactive test program, which can be started
via the 'libports/run/mupdf' run script. The program supports the basic key
handling required to browse through a multi-page PDF document
(page-up or enter -> next page, page-down or backspace -> previous page).
Improved terminal performance
=============================
The terminal component used to make all intermediate states visible to the
framebuffer in a fully synchronous fashion. This is an unfortunate behaviour
when scrolling through large text outputs. By decoupling the conversion of the
terminal state to pixels from the 'Terminal::write()' RPC function,
intermediate terminal states produced by sub sequential write operations do not
end up on screen one by one but only the final state becomes visible. This
improvement drastically improves the speed in situations with a lot of
intermediate states.
Noux support for fork semantics
===============================
Genode proclaims to be a framework out of which operating systems can be built.
There is no better way of putting this claim to the test than to use the
framework for building a Unix-like OS. This is the role of the Noux runtime
environment.
During the past releases, Noux evolved into a runtime environment that is able
to execute complex command-line-based GNU software such as VIM with no
modification. However, we cannot talk of Unix without talking about its
fundamental concept embodied in the form of the 'fork()' system call. We did
not entirely dismiss the idea of implementing 'fork()' into Noux but up to now,
it was something that we willingly overlooked. However, the primary goal of
Noux is to run the GNU userland natively on Genode. This includes a good deal
of programs that rely on fork semantics. We could either try to change all the
programs to use a Genode-specific way of starting programs or bite in the
bullet and implement fork. With the current release, we did the latter.
The biggest challenge of implementing fork was to find a solution that is not
tied to one kernel but one that works across all the different base platforms.
The principle problem of starting a new process in a platform-independent
manner is already solved by Genode in the form of the 'Process' API. But this
startup procedure is entirely different from the semantics of fork. The key to
the solution was Genode's natural ability to virtualize the access to low-level
platform resources. To implement fork semantics, all Noux has to do is to
provide local implementations of core's RAM, RM, and CPU session interfaces.
The custom implementation of the CPU session interface is used to tweak the
startup procedure as performed by the 'Process' class. Normally, processes
start execution immediately at creation time at the ELF entry point. For
implementing fork semantics, however, this default behavior does not work.
Instead, we need to defer the start of the main thread until we have finished
copying the address space of the forking process. Furthermore, we need to start
the main thread at a custom trampoline function rather than at the ELF entry
point. Those customizations are possible by wrapping core's CPU service.
The custom implementation of the RAM session interface provides a pool of RAM
shared by Noux and all Noux processes. The use of a shared pool alleviates the
need to assign RAM quota to individual Noux processes. Furthermore, the custom
implementation is needed to get hold of the RAM dataspaces allocated by each
Noux process. When forking a process, the acquired information is used to
create a shadow copy of the forking address space.
Finally, a custom RM service implementation is used for recording all RM
regions attached to the region-manager session of a Noux process. Using the
recorded information, the address-space layout can then be replayed onto a new
process created via fork.
With the virtualized platform resources in place, the only puzzle piece that
is missing is the bootstrapping of the new process. When its main thread is
started, it has an identical address-space content as the forking process but
it has to talk to a different parent entrypoint and a different Noux session.
The procedure of re-establishing the relationship of the new process to its
parent is achieved via a small trampoline function that re-initializes the
process environment and then branches to the original forking point via
setjmp/longjmp. This mechanism is implemented in the libc_noux plugin.
As the immediate result of this work, a simple fork test can be executed across
all base platforms except for Linux (Linux is not supported yet). The test
program is located at 'ports/src/test/noux_fork' and can be started with the
'ports/run/noux_fork.run' script.
Furthermore, as a slightly more exciting example, there is a run script for
running a bash shell on a tar file system that contains coreutils. By starting
the 'ports/run/noux_bash.run' script, you get presented an interactive bash
shell. The shell is displayed via the terminal service and accepts user input.
It allows you to start one of the coreutils programs such as ls or cat. Please
note that the current state is still largely untested, incomplete, and flaky.
But considering that Noux is comprised of less than 2500 lines of code, we are
quite surprised of how far one can get with such little effort.
Device drivers
##############
Driver improvements to accommodate dynamic (re-)loading
=======================================================
To support the dynamic probing of devices as performed by the new d3m
component, the ATAPI and USB device drivers have been enhanced to support the
subsequent closing and re-opening of sessions.
First bits of the d3m device-driver manager
===========================================
The abbreviation d3m stands for demo device-driver manager. It is our current
solution for the automated loading of suitable drivers as needed for running
Genode from a Live CD or USB stick. Because of the current narrow focus of d3m,
it is not a generic driver-management solution but a first step in this
direction. We hope that in the long run, d3m will evolve to become a generic
driver-management facility so that we can remove one of the "D"s from its name.
In the current form d3m solves the problems of merging input-event streams,
selecting the boot device, and dealing with failing network drivers.
When using the live CD, we expect user input to come from USB HID devices or
from a PS/2 mouse and keyboard. The live system should be operational if at
least one of those sources of input is available. In the presence of multiple
sources, we want to accumulate the events of all of them.
The live CD should come in the form of a single ISO image that can be burned onto a CDROM or
alternatively copied to an USB stick. The live system should boot fine in both
cases. The first boot stage is accommodated by syslinux and the GRUB boot
loader using BIOS functions. But once Genode takes over control, it needs to
figure out on its own from where to fetch data. A priori, there is no way to
guess whether the ATAPI driver or the USB storage driver should be used.
[image d3m_what_next]
Therefore, d3m implements a probing mechanism that starts each of the drivers,
probes for the presence of a particular file on an iso9660 file system.
[image d3m_probing]
Once d3m observes a drivers that is able to successfully access the magic file,
it keeps the driver and announces the driver's service to its own parent. For
the system outside of d3m, the probing procedure is completely transparent. D3m
appears to be just a service that always provides the valid block session for
the boot medium.
[image d3m_ready]
The network device drivers that we ported from the iPXE project cover the
range of most common wired network adaptors, in particular the E1000 family.
But we cannot presume that a computer running the live system comes equipped
with one of the supported devices. If no supported network card could be
detected the driver would simply fail. Applications requesting a NIC session
would block until a NIC service becomes available, which won't happen. To
prevent this situation, d3m wraps the NIC driver and provides a dummy NIC
service in the event the drivers fails. This way, the client application won't
block infinitely but receive an error on the first attempt to use the NIC.
ACPI support
============
To accommodate kernels like Fiasco.OC or NOVA that take advantage of x86's
APIC, we have introduced a simple ACPI parser located at 'os/src/drivers/acpi'.
The server traverses the ACPI tables and sets the interrupt line of devices
within the PCI config space to the GSIs found in the ACPI tables. Internally it
uses Genode's existing PCI driver as a child process for performing PCI access
and, in turn, announces a PCI service itself.
For obtaining the IRQ routing information from the ACPI tables without
employing a full-blown ACPI interpreter, the ACPI driver uses an ingenious
technique invented by Bernhard Kauer, which is described in the following
paper:
:[http://os.inf.tu-dresden.de/papers_ps/tr-atare-2009.pdf - ATARE - ACPI Tables and Regular Expressions]:
_TU Dresden technical report TUD-FI09-09, Dresden, Germany, August 2009_
:Usage:
Start the 'acpi_drv' in your Genode environment. Do not start the 'pci_drv'
since this will be used as a slave of the 'acpi_drv'. You still must load the
'pci_drv' in your boot loader. To integrate the ACPI driver into your boot
configuration, you may take the following snippet as reference:
!<start name="acpi">
! <resource name="RAM" quantum="2M"/>
! <binary name="acpi_drv"/>
! <provides><service name="PCI"/></provides>
! <route>
! <service name="ROM"> <parent/> </service>
! <any-service> <any-child/> <parent/> </any-service>
! </route>
!</start>
:Limitations and known issues:
Currently there is no interface to set the interrupt mode for core's IRQ
sessions (e.g., level or edge triggered). This is required by Fiasco.OCs kernel
interface. We regard this as future work.
Platform support
################
Fiasco.OC microkernel
=====================
The support for the Fiasco.OC base platform is still lacking proper handling
for releasing resources such as kernel capabilities. Although this is a known
issue, we underestimated the reach of the problem when Genode's signal API is
used. Each emitted signal happens to consume one kernel capability within core,
ultimately leading to a resource outage when executing signal-intensive code
such as the packet-stream interface. The current release comes with an interim
solution. To remedy the acute problem, we extended the 'Capability_allocator'
class with the ability to register the global ID of a Genode capability so
that the ID gets associated with a process-local kernel capability. Whenever
a Genode capability gets unmarshalled from an IPC message, the
capability-allocator is asked, with the global ID as key, whether the
kernel-cap already exists. This significantly reduces the waste of
kernel-capability slots.
To circumvent problems of having one and the same ID for different kernel
objects, the following problems had to be solved:
* Replace pseudo IDs with unique ones from core's badge allocator
* When freeing a session object, free the global ID _after_ unmapping
the kernel object, otherwise the global ID might get re-used in some
process and the registry will find a valid but wrong capability
for the ID
Because core aggregates all capabilities of all different processes, its
capability registry needs much more memory compared to a regular process.
By parametrizing capability allocators differently for core and non-core
processes, the global memory overhead for capability registries is kept
at a reasonable level.
Please note that this solution is meant as an interim fix until we have
resolved the root of the problem, which is the proper tracking and releasing
of capability selectors.
Linux
=====
Linux is one of the two original base platforms of Genode. The original
intension behind supporting Linux besides a microkernel was to facilitate
portability of the API design and to have a convenient testing environment for
platform-independent code. Running Genode in the form of a bunch of plain Linux
processes has become an invaluable feature for our fast-paced development.
To our delight, we lately discovered that the use of running Genode on Linux
can actually go far beyond this original incentive. Apparently, on Linux, the
framework represents an equally powerful component framework as on the other
platforms. Hence, Genode has the potential to become an attractive option for
creating complex component-based user-level software on Linux.
For this intended use, however, the framework has to fulfill the following
additional requirements:
* Developers on Linux expect that their components integrate seamlessly with
their existing library infrastructure including all shared libraries
installed on Linux.
* The use of a custom tool chain is hard to justify to developers who regard
Genode merely as an application framework. Hence, a way to use the normal
tool chain as installed on the Linux host system is desired.
* Application developers are accustomed with using GDB for debugging and expect
that GDB can be attached to an arbitrary Genode program in an intuitive way.
Genode's original support for Linux as base platform did not meet those
expectations. Because Genode's libc would ultimately collide with the Linux
glibc, Genode is built with no glibc dependency at all. It talks to the kernel
directly using custom kernel bindings. In particular, Genode threads are created
directly via the 'clone()' system call and thread-local storage (TLS) is managed
in the same way as for the other base platforms. This has two implications.
First, because Genode's TLS mechanism is different than the Linux TLS
mechanism, Genode cannot be built with the normal host tool chain. This
compiler would generate code that would simply break on the first attempt to
use TLS. We solved this problem with our custom tool chain, which is configured
for Genode's needs. The second implication is that GDB is not able to handle
threads created differently than those created via the pthread library. Even
though GDB can be attached to each thread individually, the debugger would not
correctly handle a multi-threaded Genode process as a multi-threaded Linux
program. With regard to the use of Linux shared libraries, Genode used to
support a few special programs that used both the Genode API and Linux
libraries. Those programs (called hybrid Linux/Genode programs) were typically
pseudo device drivers that translate a Linux API to a Genode service. For
example, there exists a framebuffer service that uses libSDL as back end.
Because those programs were a rarity, the support by the build system for such
hybrid targets was rather poor.
Fortunately, all the problems outlined above could be remedied pretty easily.
It turns out that our custom libc is simply not relevant when Genode is
used as plain application framework on Linux. For this intended use, we always
want to use the host's normal libc. This way, the sole reason for using plain
system calls instead of the pthread library vanishes, which, in turn,
alleviates the need for a custom tool chain. Genode threads are then simply
pthreads compatible with the TLS code as emitted by the host compiler and
perfectly recognised by GDB. With the surprisingly little effort of creating a
new implementation of Genode's thread API to target pthreads instead of using
syscalls, we managed to provide a decent level of support for using Genode as
user-level component framework on Linux.
These technical changes alone, however, are not sufficient to make Genode
practical for real-world use. As stated above, the few hybrid Linux/Genode
programs used to be regarded as some leprous artifacts. When using Genode as
Linux application framework, however, this kind of programs are becoming the
norm rather than an exception. For this reason, we introduced new support for
such hybrid programs into the build system. By listing the 'lx_hybrid'
library in the 'LIBS' declaration of a target, this target magically becomes a
hybrid Linux/Genode program. It gets linked to the glibc and uses pthreads
instead of direct syscalls. Furthermore, host libraries can be linked to the
program by stating their respective names in the 'LX_LIBS' variable. For an
example, please refer to the libSDL-based framebuffer at
'os/src/drivers/framebuffer/sdl/target.mk'.
To enforce the build of all targets as hybrid Linux/Genode programs, the build
system features the 'alyways_hybrid' 'SPEC' value. To make it easy to create a
build directory with all targets forced to be hybrid, we have added the new
'lx_hybrid_x86' platform to the 'create_builddir' tool.
OKL4
====
:Sending an invalid-opcode exception IPC on OKL4:
When an invalid opcode gets executed, OKL4 switches to the kernel debugger
console instead of sending an exception IPC to the userland. We fixed this
problem by removing the code that invokes the debugger console from the kernel.
:Hard-wire OKL4 elfweaver to Python 2:
Recent Linux distributions use Python version 3 by default. But OKL4's
elfweaver is not compatible with this version of Python. For this reason, we
introduced a patch for pinning the Python version used by elfweaver to
version 2.
Both patches get automatically applied when preparing the 'base-okl4'
repository via 'make prepare'.
Build system and tools
######################
:Facility for explicitly building all libraries:
During its normal operation, the build system creates libraries as mere side
effects of building targets. There is no way to explicitly trigger the build of
libraries only. However, in some circumstances (for example for testing the
thorough build of all libraries), a mechanism for explicitly building libraries
would be convenient. Hence we introduced this feature in the form of the pseudo
target located at 'base/src/lib/target.mk'. By issuing 'make lib' in a build
directory, this target triggers the build of all libraries available for the
given platform.

1029
doc/release_notes-12-05.txt
View File

File diff suppressed because it is too large Load Diff

665
doc/release_notes-12-08.txt
View File

@@ -1,665 +0,0 @@
===============================================
Release notes for the Genode OS Framework 12.08
===============================================
Genode Labs
With Genode 12.08, the project focused on platform support. It enters the world
of OMAP4-based ARM platforms, revived and vastly enhanced the support for the
NOVA hypervisor, and becomes able to run directly on ARM platforms without the
need for an underlying kernel.
The new 'base-hw' platform is a deviation from Genode's traditional approach to
complement existing kernels with user-land infrastructure. It completely leaves
the separate kernel out of the picture and thereby dwarfs the base line of the
trusted computing base of Genode-based systems to approximately the half. The
new base platform is described in Section [Genode on naked ARM hardware].
Speaking of base platforms, we are happy to have promoted the NOVA hypervisor
to a first-class citizen among the base platforms. During the last months, this
kernel underwent fundamental changes regarding its mode of development and its
feature set. This prompted us to vastly improve Genode's support for this
platform and leverage its unique features. If considering the use of Genode on
x86-based hardware, NOVA has become a very attractive foundation. Section
[Embracing the NOVA Hypervisor] describes the NOVA-specific changes.
The improvement of platform support with the current release does not entail
the base platforms only but extends to profound additions of device drivers, in
particular for the ARM-based OMAP4 SoC as used on the popular Pandaboard. We
are proud to announce the availability of device drivers for HDMI output,
SD-card, USB HID, and networking for this platform.
Beyond the low-level platform improvements, the new version comes with several
new services, optimizations of existing components, and new ported libraries.
In particular, the Noux runtime has reached a point where we can principally
execute serious networking applications such as the Lynx web browser natively
on Genode. Another example is the new FFAT-based file-system service, which
makes persistent storage available via Genode's file-system interface. By
combining this new service with existing components such as the partition
service, Noux, or the file-system plugin of the libc, a lot of new application
scenarios become available. Thanks to these new components, the framework has
become able to perform on-target debugging via GDB running in Noux, or host
the genode.org website via the lighttpd web server,
:What about the road map?:
Those of you who track the milestones laid out in our [http:/about/road-map - road map]
may wonder how Genode 12.08 relates to the stated goals. In fact, several
points of the road map haven't received the attention as originally planned.
As an explanation, let us quote the paragraph right atop of the road-map page:
"The road map is not fixed. If there is commercial interest of pushing the
Genode technology to a certain direction, we are willing to revisit our plans."
Well, this is what happened. So we traded the work on the tiled window manager,
the Intel wireless driver, and SMP support for the work on the platform topics
outlined above. Nevertheless, we stick to our overall plan to turn Genode into
a general-purpose OS that is fit for use by its developers by the end of the
year. If looking closely at the additions that come with the current release,
it will become apparent how well they fit into the big picture.
Genode on naked ARM hardware
############################
One of Genode's most distinguishing properties is the ability to use the framework
on top of a range of different kernels. This way, users of the framework
benefit from the wide variety of features provided by those kernels while
only dealing with a single API and configuration concept. For example, we
frequently find ourselves using the Linux kernel as base platform while
developing services, interfaces, and protocol stacks. By being able to start
Genode as a regular program, we effectively eliminate the reboot-time for each
test run and enjoy using commodity debugging and profiling tools. On the other
hand, if high security is a concern, NOVA and Fiasco.OC provide
capability-based security at kernel-level. So the use of one of those kernels
is desirable. Genode allows for switching between those vastly different
kernels almost seamlessly.
In general, a Genode system consists of a kernel, Genode's core, and the
largely generic components on top of core. Core abstracts away the
peculiarities of the respective kernel and provides a unified API to the
components on top. From the application's point of view both kernel and core
are always at the root of the process tree and thereby are a inherent part of
the application's trusted computing base (TCB). The distinction of both
programs is almost superficial.
Since both the kernel and core must be ultimately trusted, the complexity of
both programs is critical for each Genode-based system. On our quest for
minimizing the TCB complexity so far, however, we did not question the role of
the kernel as an inherent part of the TCB and focused our attention to the
software stack on top. However, with more and more kernels entering the
picture, we identified that there is typically a considerable overlap in
functionality between kernel and core. For example, both need to know about
address spaces and their relationship to physical memory objects. Most kernels
keep track of memory mappings in an in-kernel database. Core also needs to keep
track of this information. Consequently, we found several information
replicated without a clear benefit. With this comes a seemingly significant
redundancy of code for data structures, allocators, and utility functions.
Furthermore, there exists a class of problems that must be solved by the kernel
and core alike. In particular the resource management of dynamically allocated
in-kernel objects respectively in-core objects. Whereas core uses Genode's
resource-trading concept to solve this problem, most kernels lack a good
solution for the management of in-kernel resources and are consequently prone
to resource exhaustion problems.
Out of these observations, the idea was born to explore the opportunities of
merging both programs into one and thereby eliminating the redundancies. Our
first attempt to go into this direction was the 'base-mb' platform, which
enabled us to run Genode on the Xilinx MicroBlaze softcore CPU. With this
experiment, we gained confidence that the approach is generally feasible. So we
took on the challenge to implement the idea of a hybrid kernel/core on a more
complex architecture namely ARM Cortex-A9.
The 'base-hw' platform introduced with the current release is the intermediate
result of our experiment. With this base platform, core plays the role of core
and the kernel within one program. A few code paths that require execution in
privileged mode are executed in kernel mode whereas most code paths are
executed in user mode. Both user mode code and kernel mode code run in the same
address space. The kernel portion merely provides a few basic mechanisms
without performing complex operations such as dynamic memory allocations. For
example, if core is requested to create a new thread via core's CPU session
interface, the user-level code within core allocates a KTCB (kernel thread
control block) and UTCB (user-level thread-control block) from the physical
memory allocator and passes both physical addresses to the kernel function that
spawns the actual thread. This way, we can employ Genode's resource-trading
concept for managing typical kernel resources.
The experiment turned out to be a great success. Traditionally, we would account
at least 10,000 lines of code (LOC) for the kernel. Most kernels are actually
much larger than that. Core comes at a complexity of another 10,000 LOC. So
both kernel and core make up a base line of TCB complexity of more than 20,000
LOC. By co-locating core with the kernel, we end up with a program of just
about 13,000 LOC. The vast reduction of TCB complexity compared to having
kernel and core as separate programs strikes us.
The 'base-hw' version of core supports the complete Genode API covering support
for user-level device drivers, synchronous RPCs, asynchronous notifications,
shared memory, and managed dataspaces. It is thereby able to execute the
sophisticated Genode scenarios on top including the GUI, the dynamic linker,
and user-level device drivers. That said, we regard the current version still
as work in progress. We successfully use it as an experimentation platform for
ongoing research activities (i.e., for exploring ARM TrustZone) but some
important features such as capability-based security are not yet implemented.
:Using the base-hw platform:
The new base platform is fully integrated with Genode's build system.
When listing the supported base platforms via the 'tool/create_builddir' tool,
you will see the new 'hw_panda_a2', 'hw_vea9x4', 'hw_pbxa9' choices of
build-directory templates. The latter platform enables you to run a
'base-hw' Genode system on Qemu.
[http://genode.org/documentation/platforms/hw - Learn more about using the new base-hw platform...]
For running Genode directly on the Pandaboard, please refer to the
[http://genode.org/documentation/platforms/hw_panda_a2 - Pandaboard-specific documentation...]
Embracing the NOVA Hypervisor
#############################
NOVA is a so-called microhypervisor for the x86 architecture. It combines the
principles of microkernels with capability-based security and hardware-assisted
virtualization. Among the various base platforms supported by Genode, NOVA's
kernel interface stands out for being extremely minimalistic and orthogonal,
even by microkernel standards.
Genode has supported NOVA as base platform since 2010. But because we used NOVA
solely for sporadic research activities and NOVA's lack of a regular release
schedule, the framework's platform support received only little attention. This
has changed now. NOVA's main developer Udo Steinberg moved from TU Dresden to
Intel Labs where he leads the development of NOVA as a true Open-Source
project. In fact, the code is now being hosted at GitHub:
:[https://github.com/IntelLabs/NOVA]:
NOVA hypervisor at GitHub
Since its move to GitHub, the hypervisor has already seen significant
improvements. The repository is continuously updated, which enables us to stay
in a close feedback loop with the NOVA developers. This change of how NOVA's
development is conducted ignited our renewed interest in promoting this
platform to a first-level citizen of our framework. The first noteworthy
improvement is the recently added 64-bit support of NOVA. We enabled Genode to
work with both variants of the kernel - 32 bit and 64 bit.
But this was just the first step. The second major change addresses the
allocation of kernel resources. Early versions of the hypervisor allowed each
process to create kernel objects and thereby indirectly consume the limited
memory resources of the kernel. This is perfectly fine for a research project
but it becomes a potential denial-of-service problem in real-world use cases.
For this reason, newer versions introduced the ability to retain the allocation
of kernel objects within a trusted component only. In the Genode world, this
component is naturally core. Even though NOVA still lacks a flexible concept for
kernel-resource management as of now, Genode has become able to use NOVA
without suffering the inherent resource management limitation. This is achieved
because core is able to arbitrate the allocation of kernel resources.
The third fundamental change is the abolishment of the last traces of global
names in a NOVA-based Genode system. There are no thread IDs, object IDs, or
any other kind of globally meaningful names. Each process has a local view on
(a small part of) the system only. If a process interacts with another process,
the kernel translates the references to remote objects from one namespace to
the other. The security implications are eminent. First, a process can only
interact with or refer to objects for which it has a name, which vastly reduces
problems of ambient authority. Second, because the kernel translates names, it
becomes impossible to forge object identities. If a process tried to pass a
forged object reference to another process, the translation would simply fail,
rendering the attack ineffective.
The described changes do not come without issues, though. To make the NOVA
kernel fit with Genode's requirements, minor patches of the hypervisor are
needed. The patches are located at 'base-nova/patches/'. However, those patches
are meant as interim solutions until we find mechanisms that fit well with the
design of the hypervisor and also fulfil our requirements.
So far, we greatly enjoyed the revived collaboration with the NOVA developers
and congratulate Udo Steinberg for the new mode of development of the
hypervisor.
Base framework
##############
In the following, we describe changes of the base API that may affect users of
the framework.
:Allocation of DMA buffers:
We extended the RAM session interface with the ability to allocate DMA buffers.
The client specifies the type of RAM dataspace to allocate via the new 'cached'
argument of the 'Ram_session::alloc()' function. By default, 'cached' is true,
which corresponds to the common case and the original behavior. When setting
'cached' to 'false', core takes the precautions needed to register the memory
as uncached in the page table of each process that has the dataspace attached.
Currently, the support for allocating DMA buffers is implemented for Fiasco.OC
only. On x86 platforms, it is generally not needed. But on platforms with more
relaxed cache coherence (such as ARM), user-level device drivers should always
use uncacheable memory for DMA transactions.
:MMIO framework improvements:
As we find ourselves increasingly using the 'Register' and 'Mmio' templates
provided by 'util/register.h' and 'util/mmio.h' for dealing with memory-mapped
devices, we extended the utilities with support for 64-bit registers and a new
interface for polling bit states. The latter functionality is provided by the
new 'wait_for' function template. To decouple the MMIO-related utility code
from an actual timer facility, the function takes a so-called 'delayer' functor
as argument. This way the user of the MMIO framework is able to pick a timer
facility that fits best with the device.
:New 'memcpy' implementation:
The memory-copy functions provided by 'util/string.h' are extremely simple
and arguably slow, particularly on platforms where byte-wise copy operations
are not supported by the CPU (i.e., ARM). Hence, we have added a CPU-specific
memcpy function ('memcpy_cpu') to 'cpu/string.h', which enables us to
provide optimized implementations. So far, we did so for the ARM architecture.
Low-level OS infrastructure
###########################
FFat-based file-system service
==============================
With the previous release, we introduced Genode's file-system interface
accompanied with a simple in-memory file-system service. With the addition of
'ffat_fs', the current release adds the first persistent file system to the
framework. The service is located at 'libports/src/server/ffat_fs'. It uses
Genode's 'Block::Session' interface as back end. Therefore, it can be combined
with any of Genode's block-device drivers and the partition service called
'part_blk'. To see the new 'ffat_fs' service in action, please refer to the new
'libports/run/libc_ffat_fs.run' script.
On the course of our work on the 'ffat_fs' service, we enabled support for long
file names in libffat and added 'lseek' support to the 'libc_ffat' plugin.
TAR-based file-system service
=============================
The new 'tar_fs' service located at 'os/src/server/tar_fs' provides a read-only
file-system session interface by reading data from a TAR archive, which, in
turn, is fetched from a ROM service. By combining 'tar_fs' with the 'libc_fs'
plugin, it becomes easy to provide customized pseudo file systems to individual
Genode programs. For example, one instance of 'tar_fs' containing a static
website and a web-server configuration can be provided as file system to a web
server. The configuration is similar to the patterns known from the 'tar_rom'
and 'ram_fs' servers:
! <config>
! <archive name="tar_archive.tar" />
! <policy label="label_of_client" root="/rootdir/for/client" />
! </config>
The policy node allows for assigning different parts of one TAR archive to
different clients. For a practical usage example of 'tar_fs', please refer to
the 'libports/run/libc_fs_tar_fs.run' script.
Terminal improvements
=====================
Our work on running a growing number of command-line-based Unix programs via
Noux prompted us to improve our terminal implementation as needed. To ease
debugging for terminal colors, we changed the previous default color scheme to
fully saturated combinations of red, green, and blue. Albeit this looks quite
painful on the eyes, it is easier to spot wrong colors when using a program
that uses ncurses, for example Lynx. Furthermore, we added the handling of
sgr0 and sgr escape sequences and thereby enabled Lynx to become almost
usable when running within Noux.
Terminal cross-link service
===========================
The 'Terminal::Session' interface gets increasingly popular within Genode.
It is used by the UART drivers, the graphical terminal, GDB monitor, the TCP
terminal, and Noux. For most of these programs, their respective client or
server role is quite clear but we find ourselves tempted to combine components
in unusual ways. For example, to let Noux run an instance of GDB, which operates
on a terminal via a virtual character device. For remote debugging, GDB plays
the role of a terminal client and the UART driver plays the role of the server.
But when running GDB monitor on the same machine, GDB monitor will also
expect to play the role of the client. In order to connect GDB monitor
to a local instance of GDB, both of them being terminal clients, we need an
adapter component. This is where the new terminal cross-link service enters
the picture. It plays the role of a terminal server between exactly two
clients. The output of one client ends up as input to the other and vice
versa. Data sent to the server gets stored in a buffer of 4096 bytes (one
buffer per client). As long as the data to be written fits into the buffer, the
'write()' call returns immediately. If no more data fits into the buffer, the
'write()' call blocks until the other client has consumed some of the data from
the buffer via the 'read()' call. The 'read()' call never blocks. A signal
receiver can be used to block until new data is ready for reading.
The new terminal crosslink can be tested via the 'os/run/terminal_crosslink.run'
script. It is also used for the just mentioned on-target debugging scenario
demonstrated by the 'ports/run/noux_gdb.run' script.
DMA-aware and optimized packet streams
======================================
Motivated by our work on OMAP4 platform support, we introduced API extensions
for handling of DMA buffers to the following interfaces:
:'Attached_ram_dataspace':
The convenience utility for allocating and locally mapping a RAM dataspace
has been enhanced with the 'cached' constructor argument, which is true
by default. When using 'Attached_ram_dataspace' for allocating DMA buffers,
this argument should be set to false.
:Block and network packet stream:
The 'Block::Session' and 'Nic::Session' interfaces use Genode's packet stream
facility for transferring bulk payload between processes. A packet stream
combines shared memory with asynchronous notifications and thereby facilitates
the use of batched packet processing. To principally enable zero-copy semantics
for device drivers, the packet-stream buffer is now explicitly allocated as DMA
buffer. This clears the way to let the SD-card driver direct DMA transactions
right into the packet stream buffer. Consequently, when attaching the SD-card
driver directly to a file system, there is no copy of payload needed.
The 'Nic::Session' interface has further been improved by using a fast
bitmap allocator for allocations within the packet-stream buffer. This is
possible because networking packets have the MTU size as an upper limit.
In contrast to the 'Block::Session' interface where requests are relatively
large, 'Nic::Session' packets are tiny, and thus, greatly benefit from the
optimized allocator.
Libraries and applications
##########################
C runtime
=========
:File I/O:
We complemented our C runtime with support for the 'pread', 'pwrite', 'readv',
and 'writev' functions. The 'pread' and 'pwrite' functions are shortcuts for
randomly accessing different parts of a file. Under the hood, the functions are
implemented via 'lseek' and 'read/write'. To provide the atomicity of the
functions, a lock guard prevents the parallel execution of either or both
functions if called concurrently by multiple threads. The 'readv' and 'writev'
functions principally enable the chaining of multiple I/O requests.
Furthermore, we added 'ftruncate', 'poll', and basic support for (read-only)
mmapped files to the C runtime.
:Libc RPC framework headers:
Certain RPC headers of the libc are needed for compiling 'getaddrinfo.c'.
Unfortunately that means we have to generate a few header files, which we do
when we prepare the libc.
New and updated 3rd-party libraries
===================================
:Expat:
[http://expat.sourceforge.net - Expat] is an XML parsing library. The port of
this library was motivated by our goal to use the GNU debugger for on-target
debugging. GDB depends on this library.
:MPC and GMP:
We complemented our existing port of the
[http://gmplib.org - GNU multiple precision arithmetic library (libgmp)] with
support for the x86_64 and ARM architectures. This change combined with the
port of the [http://www.multiprecision.org/index.php?prog=mpc - MPC library]
enables us to build the Genode tool chain for these architectures.
:OpenSSL:
Our port of OpenSSL has been updated to version 1.0.1c. Because libcrypto
provides certain optimized assembler functions, which unfortunately are not
expressed with position-independent code, we removed this assembler code and
build libcrypto with '-DOPENSSL_NO_ASM'. Because the assembler code is not
needed anymore, its generation is also removed from 'openssl.mk'.
:Light-weight IP stack (lwIP):
We enabled the lwIP TCP/IP stack for 64-bit machines and updated the library to
version 1.4.1-rc1. With the new version, the call of 'lwip_loopback_init' is
not needed anymore because lwIP always creates a loopback device. Hence, we
will be able to remove the 'libc_lwip_loopback' in the future. For now, we keep
it around so we currently do not need to update the other targets that depend
on it.
:PCRE:
[http://www.pcre.org/ - PCRE] is a library for parsing regular rexpressions. We
require this library for our ongoing work on porting the lighttpd webserver.
Lighttpd web server
===================
The [http://www.lighttpd.net/ - Lighttpd] web server has been added to the
'ports' repository. The port runs as a native Genode application and ultimately
clears the way to hosting the genode.org website on Genode. To test drive this
scenario, please give the 'ports/run/genode_org.run' script a try.
At the current stage, the port is still quite limited. For example, it does not
make use of non-blocking sockets yet. But the 'genode_org.run' run script
already showcases very well how simple a Genode-based web-server appliance can
look like.
Device drivers
##############
OMAP4 platform drivers
======================
:HDMI output:
The new HDMI driver at 'os/src/drivers/framebuffer/omap4' implements Genode's
'Framebuffer::Session' interface by using the HDMI output of OMAP4. The current
version sets up a fixed XGA screen mode of 1024x768 with the RGB565 pixel
format.
:SD-card:
The new SD card driver at 'os/src/drivers/sd_card/omap4' allows the use of a
HDSD card with the Pandaboard as block service. The driver can be tested using
the 'os/run/sd_card.run' script. Because it implements the generic
'Block::Session' interface, it can be combined with a variety of other
components such as 'part_blk' (for accessing individual partitions) or
'ffat_fs' for accessing a VFAT file system on the SD card.
The driver uses the master DMA facility of the OMAP4 SD-card controller, which
yields to good performance at low CPU utilization. The throughput matches (and
in some cases outperforms) the Linux kernel driver. In the current version,
both modes of operation PIO and DMA are functional. However, PIO mode is
retained for benchmarking purposes only and will possibly be removed to further
simplify the driver.
:USB HID:
The OMAP4-based Pandaboard relies on USB for attaching input devices.
Therefore, we need a complete USB stack to enable the interactive use of the
board. Instead of implementing a USB driver from scratch, we built upon the USB
driver introduced with the Genode release 12.05. This driver was ported from the
Linux kernel.
:Networking:
The Pandaboard realizes network connectivity via the SMSC95xx chip attached to
the USB controller. Therefore, we enhanced our USB driver with support for USB
net and the smsc95xx driver. In addition to enabling the actual device-driver
functionality, the USB stack has received much attention concerning performance
optimizations. To speed up the allocation of SKBs, we replaced the former
AVL-tree based allocator with a fast bitmap allocator. For anonymous
allocations, we introduced a slab-based allocator. Furthermore, we introduced
the distinction between memory objects that are subjected to DMA operations
from non-DMA memory objects. The most profound conceptual optimization is the
use of transmit bursts by the driver. The Linux kernel, which our driver
originates from, does not provide an API for transmitting multiple packets as a
burst. For our driver, however, this optimization opportunity opened up thanks
to Genode's packet stream interface, which naturally facilitates the batching
of networking packets. So the driver has all the information needed to create
burst transactions.
USB driver
==========
By testing our new USB driver on a variety of real PC hardware, we discovered
several shortcomings, which we resolved. In particular, we added support for
more than one UHCI controller, make sure that the 'PIRQ' bit in the legacy
support register (PCI config space) of the UHCI controller is enabled and that
the 'Trap on IRQ' bit is disabled.
With those modifications in place, the USB driver works reliably on the tested
platforms.
Runtime environments
####################
Noux
====
Noux enables the easy reuse of unmodified GNU software on Genode by providing
a Unix-like kernel interface as user-level service. Because Noux is pivotal for
our plan to use Genode for productive work, we significantly enhanced and
complemented its feature set.
:Noux on ARM and x86_64:
For keeping the scope of the development manageable, the initial version of
Noux was tied to the x86_32 platform. This was not a principal limitation of
the approach but rather an artificial restriction to keep us focused on
functionality first. Now that Noux reaches a usable state, we desire to use it
on platforms other than x86_32. The current release enables Noux for the 64-bit
x86 and ARM architectures.
The level of support is pretty far-reaching and even includes the building and
execution of the Genode tool chain on those platforms. In the process of
enabling these platforms, we updated the Noux package for GCC to version 4.6.1,
which matches the version of the current Genode tool chain.
:Terminal file system:
Noux supports the concept of stacked file systems. The virtual file system
is defined at the start of a Noux instance driven by the static Noux
configuration. This way, arbitrary directory structures can be composed out
of file-system sessions and TAR archives. The VFS concept allows for the
easy addition of new file system types. To allow programs running in a Noux
instance to communicate over a dedicated terminal session, we added a new
file-system type that corresponds to a virtual character device node attached
to a terminal session.
:GDB running in the Noux environment:
With the terminal file system in place, we are ready to execute GDB within
Noux and let it talk to a GDB monitor instance over the terminal session
interface. From GDB's point of view, the setup looks like a remote debugging
session. But in reality both the debugging target and GDB reside in different
subtrees of the same Genode system.
:Executing shell scripts:
By inspecting the program specified to the execve system call, Noux has become
able to spawn scripts that use the '#!' syntax. If such a file is detected, it
executes the specified interpreter instead and passes the arguments specified
after the '#!' marker, followed by command-line arguments.
:Networking support:
Our work on porting various networking tools to Noux triggers us to steadily
improve the networking support introduced with Genode 12.05. In particular, we
added proper support for DNS resolving, which enables us to execute the
command-line based Lynx web browser within Noux.
:User information:
Because there are certain programs, which need the information that is stored
in 'struct passwd', we introduced configurable user information support to
Noux. One can set the user information via the '<user>' node in the Noux
config:
! <config>
! <user name="baron" uid="1" gid="1">
! <shell name="/bin/bash" />
! <home name="/home" />
! </user>
! ...
! </config>
When '<user>' is not specified, default values are used. Currently these
are 'root', 0, 0, '/bin/bash', '/'. Note that this is just a single user
implementation because each Noux instance has only one user or rather one
identity and there will be no complete multi-user support in Noux. If you need
different users, just start new Noux instances for each of them.
:New '/dev/null' and '/dev/zero' pseudo devices:
These device are mandatory for most programs (well, at least null is required
to be present for a POSIX compliant OS, which Noux is actually not). But for
proper shell-script support we will need them anyway. Under the hood, both
pseudo devices are implemented as individual file-systems and facilitate Noux's
support for stacked file systems. The following example configuration snippet
creates the pseudo devices under the '/dev' directory.
! <config>
! <fstab>
! <dir name="dev" >
! <null /> <zero />
! </dir>
! ...
! <fstab>
! ...
! </config>
Vancouver
=========
The comprehensive rework of the NOVA base platform affected the Genode version
of the Vancouver virtual machine monitor as this program used to speak directly
to the NOVA kernel. Since no kernel objects can be created outside of core
anymore, the Vancouver port had to be adjusted to solely use Genode interfaces.
L4Linux
=======
To improve the stability and performance of L4Linux on OMAP4 platforms, we
reworked parts of the Genode-specific stub drivers, in particular the
networking code. Among the improvements are the use of a high-performance
allocator for networking packets, improved IRQ safety of IPC calls (to
the Genode world), and tweaks of the TCP rmem and wmem buffer sizes to
achieve good TCP performance when running Linux with little memory.
Furthermore, we added two ready-to-use run scripts residing within
'ports-foc/run' as examples for executing L4Linux on the OMAP4-based
Pandaboard. The 'linux_panda.run' script is meant as a blue print for
experimentation. It integrates one instance of L4Linux with the native SD-card
driver, the HDMI driver, and the USB HID input driver. The
'two_linux_panda.run' script is a more elaborative example that executes two
instances of L4Linux, a block-device test, and a simple web server. Each of
the L4Linux instances accesses a different SD-card partition whereas the
block-device test operates on a third partition.

735
doc/release_notes-12-11.txt
View File

@@ -1,735 +0,0 @@
===============================================
Release notes for the Genode OS Framework 12.11
===============================================
Genode Labs
The central theme of version 12.11 of the Genode OS Framework is
self-hosting Genode on Genode. With self-hosting, we understand the execution of the
entire Genode build system within the Genode environment. There are two motivations
for pursing this line of work. First, it is a fundamental prerequisite for the
Genode developers to move towards using Genode as a day-to-day OS. Of course,
this prerequisite could be realized using one of the available virtualization
solutions. For example, we could run L4Linux on top of Genode on the Fiasco.OC
kernel and use the Genode build system from within an L4Linux instance.
However, this defeats the primary incentive behind Genode to reduce system
complexity. By having both Genode and L4Linux in the picture, we would indeed
increase the overall complexity in configuring, maintaining, and using the
system. Therefore, we would largely prefer to remove the complex Linux user
land from the picture. The second motivation is to prove that the framework and
underlying base platforms are suited and stable enough for real-world use.
If the system is not able to handle a workload like the build system,
there is little point in arguing about the added value of having a
microkernel-based system over current commodity OSes such as GNU/Linux.
We are happy to have reached the state where we can execute the unmodified
Genode build system directly on Genode running on a microkernel. As the
build system is based on GNU utilities and the GNU compiler collection,
significant effort went into the glue between those tools and the Genode API.
Section [Building Genode on Genode] provides insights into the way we achieved
the goal and the current state of affairs.
Along with the work on bringing the build system to Genode came numerous
stability improvements and optimizations all over the place, reaching from the
respective kernels, over the C runtime, the file-system implementations, memory
allocators, up to the actual programs the tool chain is composed of. Speaking
of the tool chain, the official Genode tool chain has been updated from GCC
version 4.6.1 to version 4.7.2. Thereby, all 3rd-party code packages were
subjected to testing and fixing activities.
For running the build system, the project currently focuses on NOVA and
Fiasco.OC as base platforms. However, our custom kernel platform for the ARM
architecture has also received significant improvements. With added support for
Freescale i.MX and Texas Instruments OMAP4, this platform proved to be very
well adaptable to new SoCs whereas new cache handling brings welcome
performance improvements. Furthermore, we have added experimental support for
ARM TrustZone technology, which principally enables the execution of Genode in
the so-called secure world of TrustZone while executing Linux in the so-called
normal world.
As we discovered the increasing interest in using Genode as a middleware
solution on Linux, we largely revisited the support for this kernel platform
and discovered amazing new ways to align the concept of Genode with the
mechanisms provided by the Linux kernel. Section [Linux] provides a summary
of the new approaches taken for supporting this platform.
Functionality-wise, the new version introduces support for audio drivers of
the Open Sound System, a new OMAP4 GPIO driver, improvements of the graphical
terminal, and the initial port of an SSH client.
Building Genode on Genode
#########################
On the Genode developer's way towards using Genode as a day-to-day OS, the
ability to execute the Genode build system within the Genode environment is a
pivotal step - a step that is highly challenging because the build system is
based on the tight interplay of many GNU programs. Among those
programs are GNU make, coreutils, findutils, binutils, gcc, and bash. Even
though there is a large track record of individual programs and libraries ported
to the environment, those programs used to be self-sustaining applications that
require only little interaction with other programs. In contrast, the build
system relies on many utilities working together using mechanisms such as
files, pipes, output redirection, and execve. The Genode base system does not
come with any of those mechanisms let alone the subtle semantics of the POSIX
interface as expected by those utilities. Being true to microkernel principles,
Genode's API has a far lower abstraction level and is much more rigid in scope.
To fill the gap between the requirements of the build system and the bare
Genode mechanisms, the Noux runtime environment was created. Noux is a Genode
process that acts like a Unix kernel. When started, it creates a child process,
which plays a similar role as the init process of Unix. This process communicates
via RPC messages to Noux. Using those messages, the process can perform all the
operations normally provided by a classical Unix kernel. When executed under
Noux, a process can even invoke functionalities such as fork and execve, which
would normally contradict with Genode's principles of resource management.
Over the course of the past year, more and more programs have been ported to
the Noux environment. Thereby, the semantics provided by Noux have been
successively refined so that those program behave as expected. This was an
iterative process. For example, at the beginning, Noux did not consider the
differences between 'lstat' and 'stat' as they did not matter for the first
batch of GNU programs ported to Noux. As soon as the programs got more
sophisticated, such shortcuts had to be replaced by the correct semantics. The
Genode build system is by far the most complex scenario exposed to Noux so far.
It revealed many shortcomings by both functionality implemented in Noux or the
C runtime as well as the underlying base platforms. So it proved to be a great
testing ground for analysing and improving those platform details. Therefore,
the secondary effects of self-hosting Genode on Genode in terms of stability
turned out to be extremely valuable.
The release comes with two ready-to-use run scripts for building bootable
system images that are able to execute the Genode tool chain, one for targeting
NOVA and one for targeting Fiasco.OC. Those run scripts are located at
'ports/run/' and called 'noux_tool_chain_nova.run' and 'noux_tool_chain_foc.run'
respectively. Each of those run scripts can be executed on either of those base
platforms. For example, by executing 'noux_tool_chain_nova' on Fiasco.OC, the
image will run Genode on Fiasco.OC and the tool chain will build binaries for
NOVA. When started, a build directory will be created at '/home/build'.
The Genode source code is located at '/genode'. In the '/bin' directory,
there are all the GNU programs needed to execute the tool chain. For
taking a look into the source code, 'vim' is available. To build core,
change to the build directory '/home/build' and issue 'make core'.
On Fiasco.OC, the complete Genode demo scenario can be compiled. On NOVA, the
incomplete life-time management of kernel objects will still result in an
out-of-memory error of the kernel. This kernel issue is currently being worked
on. Executing the tool chain on either of those platforms is still relatively
slow as extensive trace output is being generated and no actions have been taken to
optimize the performance so far. There are many opportunities for such
optimizations, which will be taken on as the next step.
Base framework
##############
Genode's base framework has received new support for extending session
interfaces and gained improvements with regard to interrupt handling on the x86
platform. At the API level, there are minor changes related to the CPU session
and 'Range_allocator' interfaces.
Support for specializing session interfaces
===========================================
With increasingly sophisticated application scenarios comes the desire to
extend Genode's existing session interface with new functionality. For example,
the 'Terminal::Session' interface covers plain read and write operations. It is
implemented by services such as a graphical terminal, the telnet-like TCP
terminal, or UART drivers. However, for the latter category, the breadth of the
interface is severely limited as UART drivers tend to supplement the read / write
interface with additional control functions, e.g., for setting the baud rate.
One way to go would be to extend the existing 'Terminal::Session' interface
with those control functions. However, these functions would be meaningless for
most implementations. Some of those other implementations may even desire their
own share of additions. In the longer term, this approach might successively
broaden the interface and each implementation will cover a subset only.
Because Genode aspires to keep interfaces as low-complex as possible while, at
the same time, it wants to accommodate the growing sophistication of usage
scenarios, we need a solution that scales. The solution turns out to be
strikingly simple. The RPC framework already supports the inheritance of RPC
interfaces. So it is possible to model the problem such that a new
'Uart::Session' interface derived from the existing 'Terminal::Session' will
be the host of UART-specific functionality. The only piece missing is the
propagation of both 'Uart' and 'Terminal' through the parent interface while
announcing the service. To spare the work of manually announcing the chain of
inherited interfaces from the implementor, the 'Parent::announce()' function has
been enhanced to automatically announce all service types implemented by the
announced interface. This way, a UART driver will always announce a "Uart"
and a "Terminal" service.
Improved interrupt handling
===========================
To accommodate modern x86 platforms, the session arguments of core's IRQ
service have been supplemented with the IRQ mode. There are two degrees of
freedom, namely the trigger (level / edge) and polarity (high / low). Thanks to
this addition, device drivers have become able to supply their knowledge of
devices to core.
In system scenarios with many peripherals, in particular when using the USB
driver, IRQ lines are shared between devices. Until now, Genode supported
shared interrupts for the OKL4 base platform only. To also cover the other
x86 kernels, we have generalized the interrupt sharing code and enabled this
feature on Fiasco.OC and NOVA.
Revised CPU session interface
=============================
We revisited the CPU session interface, removed no-longer used functions and
added support for assigning threads to CPUs.
The original CPU session interface contained functions for iterating through
the threads of a session. This interface was originally motivated by an
experimental statistical profiling tool that was developed at an early stage of
Genode. In the meanwhile, we discovered that the virtualization of the CPU
session interface is much more elegant to cover this use case than the
thread-iterator interface. Because the iteration has no transactional
semantics, it was unsafe to use it anyway.
To enable the use of multiple CPUs on multi-processor systems, the CPU
session interface has been enhanced with two functions, namely 'affinity'
and 'num_cpus'. The interface extension principally allows the assignment of
individual threads to CPUs. It is currently implemented on Fiasco.OC only.
On all other base platforms, 'num_cpus' returns one CPU. Note that on
the Linux platform, multiple CPUs will be used transparently.
The 'Cpu_session::state' function has been split into two functions, one
for retrieving information and one for propagating state information. The
prior interface was less explicit about the semantics of the 'state' function
as it took a non-const pointer to a 'Thread_state' object as argument.
Platform-tailored protection domains
====================================
Genode tries to provide a uniform API across all the different base platforms.
Yet, it also strives to make genuine platform features available to the
users of the framework. Examples for such features are the virtualization
support of the NOVA hypervisor or the special support for paravirtualizing
Linux on Fiasco.OC. Another example is the security model as found on the Linux
platform. Even though the security mechanisms of plain Linux are not as strong
as Genode's capability concept on a conceptual level, we still want to leverage
the available facilities such as user IDs and chroot as far as possible.
Consequently, we need a way to assign platform-specific properties to PD
sessions. With the new 'Native_pd_args' type introduced into
'base/native_types.h', there is now a way to express those platform-specific
concerns. This type is now used at all the places that deal with the creation
of protection domains such as 'Process', 'Child', and the loader.
Revised 'Range_allocator' interface
===================================
The handling of allocation errors has been refined in order to distinguish
different error conditions, in particular out-of-metadata and out-of-memory
conditions. The user of the allocator might want to handle both cases
differently. Hence we return an 'Alloc_return' value as result. In prior
versions, this type was just an enum value. With the new version, the type has
been changed to a class. This makes the differentiation of error conditions at
the caller side more robust because, in contrast to enum values, typed objects
don't get implicitly converted to bool values.
Low-level OS infrastructure
###########################
New UART session interface
==========================
To accommodate UART specific extensions of the 'Terminal::Session' interface,
in particular setting the baud rate, we introduced the new 'Uart::Session'
interface and changed the existing UART drivers to implement this
interface instead of the 'Terminal::Session' interface. Because 'Uart::Session'
inherits the 'Terminal::Session' interface, 'Uart' services announce both
"Uart" and "Terminal" at their parent.
New GPIO session interface
==========================
Embedded SoCs such as OMAP4 provide many general-purpose I/O pins, which can be
used for different purposes depending on the board where they are soldered on.
For example, the Pandaboard uses such GPIO pins to detect the presence of a
HDMI plug or control the power supply for the USB. If only one driver deals
with GPIO pins, the GPIO programming can reside in the driver. However, if
multiple drivers are used, the GPIO device resources cannot be handed out to
more than one driver. This scenario calls for the creation of a GPIO driver as
a separate component, which intermediates (and potentially multiplexes) the
access to the physical GPIO pins. The new 'Gpio::Session' interface allows one
or multiple clients to configure I/O pins, request states, as well as to
register for events happening on the pins.
Terminal
========
The graphical terminal has been enhanced with support for different built-in
font sizes and background-color handling.
In addition to those functional changes, the implementation has been decomposed
into several parts that thereby became reusable. Those parts comprise the
handling of key mappings, decoding the VT character stream, and the handling of
the character array. These functionalities are now available at
'gems/include/terminal'.
Libraries and applications
##########################
C runtime
=========
:Allocator optimized for small-object allocations:
To optimize the performance of workloads that depend on a large number of small
dynamic memory allocations, in particular the lwIP TCP/IP stack, we replaced
the memory allocator of the libc with a more sophisticated strategy. Until now,
the libc used 'Genode::Heap' as allocator. This implementation is an
AVL-tree-based best-fit allocator that is optimized for low code complexity
rather than performance for small allocations. The observation of the allocator
usage pattern of lwIP prompted us to replace the original libc malloc/free with
a version that uses slab allocators for small objects and relies on the
'Genode::Heap' for large objects only.
:Symbolic links:
Because part of our ongoing refinements of the Noux runtime is the provision of
symbolic links, support for symbolic links was added in the libc, libc plugins,
and file system servers.
lwIP
====
We updated the light-weight IP stack to version STABLE-1.4.1. Additionally,
the following optimizations were conducted to improve its performance and
robustness.
We reduced the maximum segment lifetime from one minute to one second to avoid
queuing up PCBs in TIME-WAIT state. This is the state, PCBs end up after
closing a TCP connection socket at the server side. The number of PCBs in this
state is apparently not limited by the value of 'MEMP_NUM_TCP_PCB'. One
allocation costs around 160 bytes. If clients connect to the server at a high
rate, those allocations accumulate quickly and thereby may exhaust the memory
of the server. By reducing the segment lifetime, PCBs in TIME-WAIT state are
cleaned up from the 'tcp_tw_pcbs' queue in a more timely fashion (by
'tcp_slowtmr()').
To prevent the TCP/IP stack from artificially throttling TCP throughput,
we adjusted lwIP's TCP_SND_BUF size.
From our work on optimizing the NIC stub-code performance of L4Linux as
described [http://genode.org/documentation/articles/pandaboard - here],
we learned that the use of a NIC-specific packet allocator for the
packet-stream interface is beneficial. At the lwIP back end, we still relied on
the original general-purpose allocator. Hence, we improved the lwIP back-end
code by using the bitmap-based 'Nic::Packet_allocator' allocator instead.
Standard C++ library
====================
Genode used to rely on the standard C++ library that comes with the tool chain.
However, this mechanism was prone to inconsistencies of the types defined in
the header files used at compile time of the tool chain and the types provided
by our libc. By building the C++ standard library as part of the Genode build
process, such inconsistencies cannot happen anymore. The current version of the
C++ standard library corresponds to GCC 4.7.2.
Note that the patch changes the meaning of the 'stdcxx' library for users that
happened to rely on 'stdcxx' for hybrid Linux/Genode applications. For such
uses, the original mechanism is still available, in the renamed form of
'toolchain_stdcxx'.
Device drivers
##############
Open Sound System
=================
Genode tries to re-use existing device drivers as much as possible using an
approach called device-driver environment (DDE). A DDE is a library that
emulates the environment of the original driver by translating device accesses
to the Genode API. There are many success stories of drivers successfully ported
to the framework this way. For example, using DDE-Linux, we are able to use the
Linux USB stack. Using DDE-ipxe, we are able to use iPXE networking drivers.
With Genode 12.11 we extend our arsenal of DDEs with DDE-OSS, which is a
device-driver environment for the audio drivers of the Open Sound System (OSS).
:Website of the Open Sound System:
[http://http://www.4front-tech.com]
The new 'dde_oss' contains all the pieces needed to use Intel HDA, AC97, and
ES1370 audio cards on Genode. On first use, the 3rd-party code can be
downloaded by issuing 'make prepare' from within the 'dde_oss' source-code
repository. Also, you need to make sure to add the 'dde_oss' repository to your
'REPOSITORIES' variable in 'etc/build.conf'.
An OSS demo configuration can be found under 'run/oss.run' and can be started
via 'make run/oss' from a Genode build directory. Be sure to adjust the
'filename' tag of the 'audio0' program. The file has to reside under
'<build-dir>/bin/'. The file format is header-less two-channel float-32 at
44100 Hz. You may use the 'sox' utility to create these audio files:
! sox -c 2 -r 44100 foo.mp3 foo.f32
OMAP4 GPIO driver
=================
The new OMAP4 GPIO driver is the first implementation of the just introduced
'Gpio::Session' interface. The driver supports two ways of interacting
with GPIO pins, by providing a static configuration, or by interacting with a
session interface at runtime. An example for a static configuration looks as
follows:
! <config>
! <gpio num="121" mode="I"/>
! <gpio num="7" mode="O" value="0"/>
! <gpio num="8" mode="O" value="0"/>
! </config>
The driver is located at 'os/src/drivers/gpio/omap4'. As reference for using
the driver, please refer to the 'os/run/gpio_drv.run' script.
Thanks to Ivan Loskutov of Ksys-Labs for contributing the session interface
and the driver!
iPXE networking drivers
=======================
We updated our device-driver environment for iPXE networking drivers to a
recent git revision and enabled support for the x86_64 architecture.
Currently, the driver covers Intel gigabit ethernet (e1000, e1000e, igb),
Intel eepro100, and Realtek 8139/8169.
Runtime environments
####################
Noux
====
The Noux runtime environment has received plenty of love thanks to the
aspiration to execute the Genode build system.
:Time:
The build system uses GNU make, which depends on time stamps of files. We do
not necessarily need a real clock. A monotonic increasing virtual time is
enough. To provide such a virtual time, the libc was enhanced with basic
support for functions like 'gettimeofday', 'clock_gettime', and 'utimes'. As
there is currently no interface to obtain the real-world time in Genode, Noux
simulates a pseudo real-time clock using a jiffies-counting thread. This
limited degree of support for time is apparently sufficient to trick tools like
ping, find, and make into working as desired.
:Improved networking support:
The Noux/net version of Noux extends the Noux runtime with the BSD-socket
interface by using the lwIP stack. This version of Noux multiplexes the
BSD-socket interface of lwIP to multiple Noux programs, each having a different
socket-descriptor name space and the principal ability to use blocking calls
such as 'select'. The code for multiplexing the lwIP stack among multiple Noux
processes has been improved to cover corner cases exposed by sophisticated
network clients, i.e., openssh.
:Directory cache for the TAR file system:
The original version of the TAR file system required a search in all TAR
records for each file lookup. This takes a long time when composing a large
directory tree out of multiple TAR archives stacked together. This is the case
for the Genode build-system scenario where we have all the files of the GNU
tools as well as the Genode source tree. Searching through thousands of records
for each call of 'stat' quickly becomes a scalability issue. Therefore, we
introduced a TAR indexing mechanism that scans each TAR file only once at the
startup of Noux and generates a tree structure representing the directory
layout. Looking up files using this index is quick.
:New packages:
With Genode-12.11, new 3rd-party packages have become available, namely
OpenSSH, the 'which' command, and all tool-chain components in their current
version. OpenSSH is still at an experimental stage. The run script at
'ports/run/noux_net_openssh_interactive.run' demonstrates how SSH can be used
to login into a remote machine.
:New pseudo file systems:
The new 'stdio' and 'random' file systems are intended to represent the pseudo
devices '/dev/random' and '/dev/tty' on Noux. Both are needed to run OpenSSH.
Note that the 'Arc4random' class, on which the random file system is based on,
currently _does not collect enough_ random bytes! It should not be used for
security-critical applications.
L4Linux
=======
The paravirtualized L4Linux kernel for the Fiasco.OC platform was updated to
SVN revision 25, which matches the Fiasco.OC SVN revision 40. We further
improved the integration of L4Linux with Genode by optimizing the stub drivers
for block devices and networking, and added principal support for running
L4Linux on SMP platforms.
Platforms
#########
NOVA
====
Genode follows the steady development of the NOVA microhypervisor very closely.
The kernel used by the framework corresponds to the current state of the master
branch of IntelLabs/NOVA.
:Improvements towards GDB support:
The NOVA-specific implementation of the CPU session interface has been improved
to accommodate the requirements posed by GDB. In particular, the 'pause',
'resume', 'state', and 'single_step' functions have been implemented. Those
functions can be used to manipulate the execution and register state of
threads. Under the hood, NOVA's 'recall' feature is used to implement these
mechanisms. By issuing a 'recall' for a given thread, the targeted thread is
forced into an exception. In the exception, the current state of the thread can
be obtained and its execution can be halted/paused.
:Maximizing contiguous virtual space:
To enable the Vancouver virtual machine monitor to hand out large amounts of
guest memory, we optimized core's virtual address space to retain large and
naturally aligned contiguous memory regions. For non-core processes, the
thread-context area that contains the stacks of Genode threads has been moved
to the end of the available virtual address space.
:Life-time management of kernel resources:
We improved the life-time management of kernel resources, in particular
capabilities, within Genode. Still the management of such kernel resources
is not on par with the Fiasco.OC version, partially because of missing
kernel functionality. This is an ongoing topic that is being worked on.
:Using the BIOS data area (BDA) to get serial I/O ports on x86:
If the I/O ports for the comport are non default (default is 0x3f8), we had to
specify manually the correct I/O ports in the source code. To avoid the need
for source-code modifications when changing test machines, we changed the core
console to read the BDA and use the first serial interface that is available.
If no serial interface is available, no device configuration will be
undertaken. The BDA can be populated via a multi-boot chain loader. Bender is
such a chain loader that can detect serial ports accessible via PCI and writes
the I/O ports to the Bios Data area (BDA). These values get then picked up by
core.
Fiasco.OC
=========
The Fiasco.OC kernel has been updated to the SVN revision 40. The update improves
SMP support and comes with various bug fixes. There is no noteworthy change
with regard to the kernel interface. We extended the number of supported
Fiasco.OC-based platforms for Genode by including the Freescale i.MX53.
To enable the use of multiple CPUs by Genode processes, the CPU session
interface has been enhanced to support configuring the affinity of threads with
CPUs. We changed the default kernel configuration for x86 and ARM to
enable SMP support and adapted L4Linux to use the new interface.
Execution on bare hardware (base-hw)
====================================
The development of our custom platform for executing Genode directly on bare
hardware with no kernel underneath went full steam ahead during the release
cycle.
:Pandaboard:
The in-kernel drivers needed to accommodate the Pandaboard, more specifically
the timer and interrupt controller, are now supported. So the Pandaboard can be
used with both 'base-hw' and 'base-foc'. Also, the higher-level platform
drivers for USB, HDMI, and SD-card that were introduced with the previous
release, are equally functional on both platforms.
:Freescale i.MX31:
We added principal support for the Freescale i.MX line of SoCs taking the
ARMv6-based i.MX31 as starting point. As of now, the degree of support is
limited to the devices needed by the kernel to operate. Pure software-based
scenarios are able to work, i.e., the nested init run script executes
successfully.
:TrustZone support:
The new VM session interface of core provides a way to execute software
in the normal world of a TrustZone system whereas Genode runs in the secure
world. From Genode's point of view, the normal world looks like a virtual
machine. Each time, the normal world produces a fault or issues a secure
monitor call, control gets transferred to the virtual machine monitor, which is
a normal user-level Genode process. The base-hw kernel has been enhanced to
perform world switches between the secure and normal world and with the ability
to handle fast interrupts (FIQs) in addition to normal interrupts. The latter
extension is needed to assign a subset of devices to either of both worlds.
Currently, the only TrustZone capable platform is the ARM CoreTile Express
CA9x4 for the Versatile Express board. For a virtual machine working properly
on top, some platform resources must be reserved. Therefore, there exist two
flavours of this platform now, one with the 'trustzone' spec-variable enabled
and one without. If 'trustzone' is specified, most platform resources (DDR-RAM,
and most IRQs) are reserved for the normal world and not available to the
secure Genode world.
:Memory attributes and caching:
We successively activated various levels of caching and improved the handling
of caching attributes propagated into the page tables. These changes resulted
in a significant boost in performance on non-emulated platforms.
Linux
=====
The Linux version of Genode was originally meant as a vehicle for rapid
development. It allows the framework components including core to be executed
as plain Linux processes. But in contrast to normal Linux programs, which
use the glibc, Genode's components interact with the kernel directly without
any C runtime other than what comes with Genode. We use the Linux version on a
regular basis to implement platform-agnostic functionality and protocols. Most
of Genode's code (except for device drivers) falls in this category. Because
the Linux version was meant as a mere tool, however, we haven't put much
thought into the principle way to implementing Genode's security concept on
this platform. Threads used to communicate over globally accessible Unix-domain
sockets and memory objects were represented as globally accessible files within
'/tmp'.
That said, even though Linux was not meant as a primary platform for Genode in
the first place, Genode can bring additional value to Linux. When considering
the implementation of a component-based system on Linux, there are several
possible approaches to take. For example, components may use DBus to
communicate, or components could pick from the manifold Unix mechanisms such as
named pipes, files, sysv-shared memory, signals, and others. Unfortunately
those mechanisms are not orthogonal and most of them live in the global name
space of the virtual file system. Whereas those mechanisms are principally able
to let processes communicate, questions about how processes get to know each
other, access-control policy, synchronization of the startup of processes are
left to the developer.
Genode, on the other hand, does provide an API for letting components
communicate but also answers those tricky questions concerning the composition
of components. This makes Genode an interesting option to build component based
applications, even on Linux. However, when used in such a context, the
limitations of the original Linux support need resolutions. Therefore, the
current release comes with a largely revised platform support for the Linux
base platform.
The changes can be summarized as follows:
:Using file descriptors as communication addresses:
Genode's synchronous RPC framework was using Unix domain sockets. Each RPC
entrypoint was represented by a pair of named files, one for sending and one
for receiving messages. In the new version, inter-process communication is
performed via file descriptors only.
:Transfer of communication rights via RPC only:
Capabilities used to be represented as a pair of the destination thread ID and
a global object ID. The thread ID has been replaced by a file descriptor that
points to the corresponding RPC entrypoint. When capabilities are transferred
as RPC arguments, those file descriptors are transferred via SCM rights
messages. This is in line with Genode's way of capability-based delegation of
access rights.
:Core-only creation of communication channels:
Communication channels used to be created locally by each process. The naming
of those channels was a mere convention. In contrast, now, communication
channels are created by core only and do not reside on the Linux virtual file
system. When creating an RPC entrypoint, core creates a socket pair and hands
out both ends to the creator of the entrypoint.
:Restricted access to memory objects:
Access to dataspace content was performed by mmap'ing a file. For a given
dataspace, the file name could be requested at core via a Linux-specific RPC
call. Now, core holds the file descriptors of all dataspaces, which are
actually unlinked files. A process that is in possession of a dataspace
capability can request the file descriptor for the content from core and mmap
the file locally. This way, access to memory objects is subjected to the
delegation of dataspace capabilities.
:Core-local process creation:
Genode used to create new processes by directly forking from the respective
Genode parent using the process library. The forking process created a PD
session at core merely for propagating the PID of the new process into core
(for later destruction). This traditional mechanism has the following
disadvantages:
First, the PID reported by the creating process to core cannot easily be
validated by core. Therefore core has to trust the PD client to not specify a
PID of an existing process, which would happen to be killed once the PD session
gets destructed. Second, there is no way for a Genode process to detect the
failure of any of its grandchildren. The immediate parent of a faulting process
could use the SIGCHLD-and-waitpid mechanism to observe its children but this
mechanism does not work transitively.
By performing the process creation exclusively within core, all Genode
processes become immediate child processes of core. Hence, core can respond to
failures of any of those processes and reflect such conditions via core's
session interfaces. Furthermore, the PID associated to a PD session is locally
known within core and cannot be forged anymore. In fact, there is actually no
need at all to make processes aware of any PIDs of other processes.
:Handling of chroot, user IDs, and group IDs:
With the move of the process creation into core, the original chroot trampoline
mechanism implemented in 'os/src/app/chroot' does not work anymore. A process
could simply escape the chroot environment by spawning a new process via core's
PD service. Therefore, chroot support has been integrated into core and the
chroot policy becomes a mandatory part of the process creation. For each process
created by core, core checks for a 'root' argument of the PD session. If a path
is present, core takes the precautions needed to execute the new process in the
specified chroot environment.
This conceptual change implies minor changes with respect to the Genode API and
the configuration of the init process. The API changes are the enhancement of
the 'Genode::Child' and 'Genode::Process' constructors to take the root path as
argument. Init supports the specification of a chroot per process by specifying
the new 'root' attribute to the '<start>' node of the process. In line with
these changes, the 'Loader::Session::start' function has been enhanced with the
additional (optional) PD argument.
In line with how the chroot path can be propagated into core, core has become
able to assign customized UIDs and GIDs to individual Genode processes or whole
Genode subsystems. The new 'base-linux/run/lx_uid.run' script contains an
example of how to use the feature.
Build system and tools
######################
The current release comes with a new tool chain based on GCC 4.7.2 and binutils
2.22. The tool-chain upgrade involved adapting the Genode code base and fixing
various issues in 3rd-party software. To obtain the new tool chain, please
refer to the tool-chain website:
:Genode tool chain:
[http://genode.org/download/tool-chain]

941
doc/release_notes-13-02.txt
View File

@@ -1,941 +0,0 @@
===============================================
Release notes for the Genode OS Framework 13.02
===============================================
Genode Labs
Traditionally, the February release of Genode is focused on platform support.
The version 13.02 follows this tradition by vastly improving Genode for the
NOVA base platform and the extending the range of ARM SoCs supported by
both our custom kernel platform and the Fiasco.OC kernel.
The NOVA-specific improvements concern three major topics, namely the added
support for running dynamic workloads on this kernel, the use of IOMMUs, and
the profound integration of the Vancouver virtual machine monitor with the
Genode environment. The latter point is particularly exciting to us because
this substantial work is the first contribution by Intel Labs to the Genode
code base. Thanks to Udo Steinberg and Markus Partheymüller for making that
possible.
Beyond the x86 architecture, the new version comes with principal support for
the ARM Cortex-A15-based Exynos 5250 SoC and the Freescale i.MX53 SoC. The
Exynos 5250 SoC has been enabled for our custom kernel as well as for the
Fiasco.OC kernel. The most significant functional improvements are a new
facility to detect faulting processes and a new mechanism for file-system
notifications.
Besides those added functionalities, the release cycle was taken as an
opportunity to revisit several aspects under the hood of the framework. A few
examples are the reworked synchronization primitives, the simplified base
library structure, the completely redesigned audio-output interface, and a
modernized timer interface.
DMA protection via IOMMU
########################
Direct memory access (DMA) of devices is universally considered as the Achilles
heel of microkernel-based operating systems. The most compelling argument in
favour of using microkernels is that by encapsulating each system component
within a dedicated user-level address space, the system as a whole becomes more
robust and secure compared to a monolithic operating-system kernel. In the
event that one component fails due to a bug or an attack, other components
remain unaffected. The prime example for such buggy components are device
drivers. By empirical evidence, those remain the most prominent trouble makers
in today's operating systems. Unfortunately however, most commodity hardware
used to render this nice argumentation moot because it left one giant loophole
open, namely bus-master DMA.
Via bus-master DMA, a device attached to the system bus is able to directly
access the RAM without involving the CPU. This mechanism is crucial for all
devices that process large amounts of data such as network adapters, disk
controllers, or USB controllers. Because those devices can issue bus requests
targeting the RAM directly and not involving the CPU altogether, such requests
are naturally not subjected by the virtual-memory mechanism implemented in the
CPU in the form of an MMU. From the device's point of view there is just
physical memory. Hence, if a driver sets up a DMA transaction, let's say a disk
driver wants to read a block from the disk, the driver tells the device about
the address and size of a physical-memory buffer where the it wants to receive
the data. If the driver lives in a user-level process, as is the case for a
microkernel-based system, it still needs to know the physical address to
program the device correctly. Unfortunately, there is nothing to prevent the
driver from specifying any physical address to the device. Consequently, a
malicious driver could misuse the device to read and manipulate all parts of
the memory, including the kernel.
[image no_iommu]
Traditional machine without IOMMU. Direct memory accesses issued by the
disk controller are not subjected to the MMU. The disk controller can
access the entity of memory present in the system.
So - does this loop hole render the micro-kernel approach useless? Of course not.
Putting each driver in a dedicated address space is still beneficial in two
ways. First, classes of bugs that are unrelated to DMA remain confined in the
driver's address space. In practice most driver issues arise from issues like
memory leaks, synchronization problems, deadlocks, flawed driver logic, wrong
state machines, or incorrect device-initialization sequences. For those classes
of problems, the microkernel argument still applies. Second, executing a driver
largely isolated from other operating-system code minimizes the attack surface
of the driver. If the driver interface is rigidly small and well-defined, it is
hard to compromise the driver by exploiting its interface.
Still the DMA issue remains to be addressed. Fortunately, modern PC hardware
has closed the bus-master-DMA loophole by incorporating a so-called IOMMU into
the system. As depicted in the following figure, the IOMMU sits between the RAM
and the system bus where the devices are attached to. So each DMA request has
to pass the IOMMU, which is not only able to arbitrate the access of DMA
requests to the RAM but also able to virtualize the address space per device.
Similar to how a MMU confines each process running on the CPU within a distinct
virtual address space, the IOMMU is able to confine each device within a
dedicated virtual address space. To tell the different devices apart, the IOMMU
uses the PCI device's bus-device-function triplet as unique identification.
[image iommu]
An IOMMU arbitrates and virtualizes DMA accesses issued by a device to the
RAM. Only if a valid IOMMU mapping exists for a given DMA access, the memory
access is performed.
Of the microkernels supported by Genode, NOVA is the first kernel that supports
the IOMMU. NOVAs interface to the IOMMU is quite elegant. The kernel simply
applies a subset of the (MMU) address space of a process (aka protection domain
in NOVA speak) to the (IOMMU) address space of a device. So the device's
address space can be managed in the same way as we normally manage the address
space of a process. The only missing link is the assignment of device address
spaces to process address spaces. This link is provided by the dedicated system
call "assign_pci" that takes a process identifier and a device identifier as
arguments. Of course, both arguments must be subjected to a security policy.
Otherwise, any process could assign any device to any other process. To enforce
security, the process identifier is a capability to the respective protection
domain and the device identifier is a virtual address where the extended PCI
configuration space of the device is mapped in the specified protection domain.
Only if a user-level device driver got access to the extended PCI configuration
space of the device, it is able to get the assignment in place.
To make NOVA's IOMMU support available to Genode programs, we enhanced the
ACPI/PCI driver with the ability to hand out the extended PCI configuration
space of a device and added a NOVA-specific extension to the PD session
interface. The new 'assign_pci' function allows the assignment of a PCI device
to the protection domain.
[image iommu_aware]
NOVAs management of the IOMMU address spaces facilities the use of
driver-local virtual addresses as DMA addresses.
Even though these mechanisms combined principally
suffice to let drivers operate with the IOMMU enabled, in practice, the
situation is a bit more complicated. Because NOVA uses the same
virtual-to-physical mappings for the device as it uses for the process, the DMA
addresses the driver needs to supply to the device must be virtual addresses
rather than physical addresses. Consequently, to be able to make a device
driver usable on systems without IOMMU as well as on systems with IOMMU, the
driver needs to be IOMMU-aware and distinguish both cases. This is an
unfortunate consequence of the otherwise elegant mechanism provided by NOVA. To
relieve the device drivers from caring about both cases, we came up with a
solution that preserves the original device interface, which expects physical
addresses. The solution comes in the form of so called device PDs. A device PD
represents the address space of a device as a Genode process. Its sole purpose
is to hold mappings of DMA buffers that are accessible by the associated
device. By using one-to-one physical-to-virtual mappings for those buffers
within the device PD, each device PD contains a subset of the physical address
space. The ACPI/PCI server performs the assignment of device PDs to PCI
devices. If a device driver intends to use DMA, it asks the ACPI/PCI driver for
a new DMA buffer. The ACPI/PCI driver allocates a RAM dataspace at core,
attaches it to the device PD using the dataspace's physical address as virtual
address, and hands out the dataspace capability to the driver. If the driver
requests the physical address of the dataspace, the returned address will be a
valid virtual address in the associated device PD. From this design follows
that a device driver must allocate DMA buffers at the ACPI/PCI server (while
specifying the PCI device the buffer is intended for) instead of using core's
RAM service to allocate buffers anonymously. The current implementation of the
ACPI/PCI server assigns all PCI devices to only one device PD. However, the
design devises a natural way to partition devices into different device PDs.
[image iommu_agnostic]
By modelling a device address space as a dedicated process (device PD),
the traditional way of programming DMA transactions can be maintained,
even with the IOMMU enabled.
Because the changed way of how DMA buffers are allocated, our existing drivers
such as the AHCI disk driver, the OSS sound driver, the iPXE network driver,
and the USB driver had to be slightly modified. We also extended DDE Kit with
the new 'dde_kit_pci_alloc_dma_buffer' function for allocating DMA buffers.
With those changes, the complete Genode user land can be used on systems with
IOMMU enabled. Hence, we switched on the IOMMU on NOVA by default.
Full virtualization on NOVA/x86
###############################
Vancouver is a x86 virtual machine monitor that is designed to run as
user-level process on top of the NOVA hypervisor. In
[http://genode.org/documentation/release-notes/11.11#Faithful_x86_PC_Virtualization_enabled_by_the_Vancouver_VMM - Genode version 11.11],
we introduced the preliminary adaptation of Vancouver to Genode. This version
was meant as a mere proof of concept, which allowed the bootup of small Guest
OSes (such as Fiasco.OC or Pistachio) inside the VMM. However, it did not
support any glue code to Genode's session interface, which limited the
usefulness of this virtualization solution at that point. We had planned to
continue the integration of Vancouver with Genode once we observed public
demand.
The move of NOVA's development to Intel Labs apparently created this demand.
It is undeniable that combining the rich user land provided by Genode with the
capabilities of the Vancouver VMM poses an attractive work load for NOVA. So
the stalled line of the integration work of Vancouver with Genode was picked up
within Intel Labs, more specifically by Markus Partheymüller. We are delighted
to be able to merge the outcome of this undertaking into the mainline Genode
development. Thanks to Intel Labs and Markus in particular for this substantial
contribution!
The features added to the new version of Vancouver for Genode are as follows:
:VMX support:
Our initial version supported AMD's SVM technology only because this was
readily supported by Qemu. With the added support for Intel VMX, Vancouver
has become able to operate on both Intel and AMD processors with hardware
virtualization support.
:Timer support:
With added support for timer interrupts, the VMM has become able to
boot a complete Linux system.
:Console support:
With this addition, the guest VM can be provided with a frame buffer and
keyboard input.
For the frame-buffer size in Vancouver, the configuration value in the
machine XML node is used. It is possible to map the corresponding memory
area directly to the guest regardless if it comes from nitpicker, a virtual
frame buffer, or the VESA driver. The guest is provided with two modes (text
mode 3 and graphics mode 0x114 (0x314 in Linux).
Pressing LWIN+END while a VM has focus resets the virtual machine. Also,
RESET and DEBUG key presses will not be forwarded to the VM anymore.
It is possible to dump a VM's state by pressing LWIN+INS keys.
The text console is able to detect idle mode, unmaps the buffer from the
guest and stops interpreting. Upon the next page fault in this area, it
resumes operation again. The code uses a simple checksum mechanism instead
of a large buffer and 'memcmp' to detect an idle text console. False
positives don't matter very much.
:Network support:
The VMM has become able to use the Intel 82576 device model from the NUL
user land to give VMs access to the network via Genode's NIC bridge service
or a NIC driver.
:Disk support:
The VMM can now assign block devices to guests using Genode's block-session
interface. The machine has to be configured to use a specified drive, which
could be theoretically routed to different partitions or services via policy
definitions. Currently the USB driver only supports one device. Genode's AHCI
driver is untested.
:Real-time clock:
By using the new RTC session interface, Vancouver is able to provide the
wall-clock time to guest OSes.
To explore the new version of the Vancouver VMM, there exists a ready-to-use
run script at 'ports/run/vancouver.run'. Only the guest OS binaries such as a
Linux kernel image and a RAM disk must be manually supplied in the
'<build-dir>/bin' directory. The run script is able to start one or multiple
instances of the VMM using the graphical launchpad.
Low-latency audio output
########################
In version 10.05, we introduced an interface for the playback of audio data
along with an audio mixer component and ALSA-based sound drivers ported from
the Linux kernel. The original 'Audio_out' session interface was based on
Genode's packet stream facility, which allows the communication of bulk data
across address spaces via a combination of shared memory and signals. Whereas
shared memory is used to transfer the payload in an efficient manner without
the need to copy data via the kernel, signals are used to manage the data flow
between the information source and sink.
[image packet_stream]
Figure [packet_stream] displays the life cycle of a packet of information
transferred from the source to the sink. The original intent behind the
packet-stream facility was the transmission of networking packets and blocks
of block devices. At the time when we first introduced the 'Audio_out'
interface, the packet stream seemed like a good fit for audio, too. However, in
the meanwhile, we came to the conclusion that this is not the case when trying
to accommodate streamed audio data and sporadic audio output at the same time.
For the output of streamed audio data, a codec typically decodes a relatively
large portion of an audio stream and submits the sample data to the mixer. The
mixer, in turn, mixes the samples of multiple sources and forwards the result
to the audio driver. Each of those components the codec, the mixer, and the
audio driver live in a separate process. By using large buffer sizes between
them, the context-switching overhead is hardly a concern. Also, the driver can
submit large buffers of sample data to the sound device without any further
intervention needed.
In contrast, sporadic sounds are used to inform the user about an immediate
event. It is ultimately expected that such sounds are played back without much
latency. Otherwise the interactive experience (e.g., of games) would suffer.
Hence, using large buffers between the audio source, the mixer, and the driver
is not an option. By using the packet stream concept, we have to settle on a
specific buffer size. A too small buffer increases CPU load caused by many
context switches and the driver, which has to feed small chunks of sample data
to the sound device. A too large buffer, however, makes sporadic sounds at low
latencies impossible. We figured out that the necessity to find a sweet spot
for picking a buffer size is a severe drawback. This observation triggered us
to replace the packet-stream-based communication mechanism of the 'Audio_out'
session interface by a new solution that we specifically designed to
accommodate both corner cases of audio output.
[image audio_out]
Similarly to the packet-stream mechanism, the new interface is based on a
combination of shared memory and signals. However, we dropped the notion of
ownership of packets. When using the packet-stream protocol depicted as above,
either the source or the sink is in charge of handling a given packet at a
given time, not both. At the points 1, 2, and 4, the packet is owned by the
source. At the points 3 and 4, the packet is owned by the sink. By putting a
packet descriptor in the submit queue or acknowledgement queue, there is a
handover of responsibility. The new interface weakens this notion of ownership
by letting the source update once submitted audio frames even after submitting
them. If there are solely continuous streams of audio arriving at the mixer,
the mixer can mix those large batches of audio samples at once and pass the
result to the driver.
[image mixer_streaming]
The mixer processes incoming data from multiple streaming sources as batches.
Now, if a sporadic sound comes in, the mixer checks the
current output position reported by the audio driver, and re-mixes those
portions that haven't been played back yet by incorporating the sporadic sound.
So the buffer consumed by the driver gets updated with new data.
[image mixer_sporadic]
A sporadic occuring sound prompts the mixer to remix packets that are
already submitted in the output queue.
Besides changing the way of how packets are populated with data, the second
major change is turning the interface into a time-triggered concept. The
driver produces periodic signals that indicate the completeness of a
played-back audio packet. This signal triggers the mixer to become active,
which in turn serves as a time base for its clients. The current playback
position is denoted alongside the sample data as a field in the memory buffer
shared between source and sink.
The new 'Audio_out' interface has the potential to align the requirements of
both streamed audio with those of sporadic sounds. As a side benefit, the now
domain-specific interface has become simpler than the original packet-stream
based solution. This becomes nowhere as evident as in the implementation of the
mixer, which has become much simpler (30% less code). The interface change
is accompanied with updates of components related to audio output, in
particular the OSS sound drivers contained in 'dde_oss', the ALSA audio driver
for Linux, the avplay media player, and the libSDL audio back-end.
Base framework
##############
Signalling API improvements
===========================
The signalling API provided by 'base/signal.h' is fairly low level. For
employing the provided mechanism by application software, we used to craft
additional glue code that translates incoming signals to C++ method
invocations. Because the pattern turned out to be not only useful but a good
practice, we added the so called 'Signal_dispatcher' class template to the
signalling API.
In addition to being a 'Signal_context', a 'Signal_dispatcher' associates a
member function with the signal context. It is intended to be used as a member
variable of the class that handles incoming signals of a certain type. The
constructor takes a pointer-to-member to the signal handling function as
argument. If a signal is received at the common signal reception code, this
function will be invoked by calling 'Signal_dispatcher_base::dispatch'. This
pattern can be observed in the implementation of RAM file system
('os/src/server/ram_fs').
Under the hood, the signalling implementation received a major improvement with
regard to the life-time management of signal contexts. Based on the observation
that 'Signal' objects are often referring to non-trivial objects derived from
'Signal_context', it is important to defer the destruction of such objects to a
point when no signal referring to the context is in flight anymore. We solved
this problem by modelling 'Signal' type as a shared pointer that operates on a
reference counter embedded in the corresponding 'Signal_context'. Based on
reference counter, the 'Signal_receiver::dissolve()' function does not return
as long as the signal context to be dissolved is still referenced by one or
more 'Signal' objects.
Trimmed and unified framework API
=================================
A though-provoking
[http://sourceforge.net/mailarchive/forum.php?thread_name=CAGQ-%3Dq27%2B_UooBiJmz9RdTE1gDmVcg9v0w-8TNgEH5fzHYiA%2BQ%40mail.gmail.com&forum_name=genode-main - posting]
on our mailing list prompted us to explore the idea to make shared libraries
and dynamically linked executables binary compatible among different kernels.
This sounds a bit crazy at first but it is not downright infeasible.
As a baby step into this direction, we unified several public headers of the
Genode API and tried to make headers private to the framework where possible.
The latter is the case for the 'base/platform_env.h' header, which is actually
not part of the generic Genode API. Hence, it was moved to the
framework-internal 'src/base/env'. Another step was the removal of
platform-specific types that are not necessarily platform-dependent. We could
remove the 'Native_lock' type without any problems. Also, we were able to unify
the IPC API, which was formerly split into the two parts 'base/ipc_generic.h'
and 'base/ipc.h' respectively. Whereas 'base/ipc_generic.h' was shared among
all platforms, the 'base/ipc.h' header used to contain platform-specific IPC
marshalling and unmarshalling code. But by moving this code from the header to
the corresponding (platform-specific) IPC library, we were able to discard the
content of 'base/ipc.h' altogether. Consequently, the former
'base/ipc_generic.h' could be renamed to 'base/ipc.h'. These changes imply no
changes at the API level.
Simplified structure of base libraries
======================================
The Genode base API used to come in the form of many small libraries, each
covering a dedicated topic. Those libraries were 'allocator_avl', 'avl_tree',
'console', 'env', 'cxx', 'elf', 'env', 'heap', 'server', 'signal', 'slab',
'thread', 'ipc', and 'lock'. The term "library" for those bits of code was
hardly justified as most of the libraries consisted of only a few .cc files.
Still the build system had to check for their inter-dependencies on each run of
the build process. Furthermore, for Genode developers, specifying the list of
base libraries in their 'target.mk' files tended to be an inconvenience. Also,
the number of libraries and their roles (core only, non-core only, shared by
both core and non-core) were not easy to capture. Hence, we simplified the way
of how those base libraries are organized. They have been reduced to the
following few libraries:
* 'cxx.mk' contains the C++ support library
* 'startup.mk' contains the startup code for normal Genode processes
On some platform, core is able to use the library as well.
* 'base-common.mk' contains the parts of the base library that are
identical by core and non-core processes.
* 'base.mk' contains the complete base API implementation for non-core
processes
Consequently, the 'LIBS' declaration in 'target.mk' files becomes simpler as
well. In the normal case, only the 'base' library must be mentioned.
New fault-detection mechanism
=============================
Until now, it was hardly possible for a parent process to respond to crashes of
child processes in a meaningful way. If a child process crashed, the parent
would normally just not take notice. Even though some special use cases such as
GDB monitor could be accommodated by the existing
'Cpu_session::exception_handler' facility, this mechanism requires the
virtualization of the 'Cpu_session interface' because an exception handler used
to refer to an individual thread rather than the whole process. For ordinary
parents, this mechanism is too cumbersome to use. However, there are several
situations where a parent process would like to actively respond to crashing
children. For example, the parent might like to restart a crashed component
automatically, or enter a special failsafe mode.
To ease the implementation of such scenarios, we enhanced the existing
'Cpu_session::exception_handler' mechanism with the provision of a
default signal handler that is used if no thread-specific handler is installed.
The default signal handler can be set by specifying an invalid thread
capability and a valid signal-context capability. So for registering a signal
handler to all threads of a process, no virtualization of the 'Cpu_session'
interface is needed anymore. The new mechanism is best illustrated by the
'os/run/failsafe.run' script, which creates a system that repeatedly spawns a
crashing child process.
Reworked synchronization primitives
===================================
We reworked the framework-internal lock interface in order to be able to use
the 'futex' syscall on the Linux base platform. Previously, the lock
implementation on Linux was based on Unix signals. In the contention case, the
applicant for a contended lock would issue a blocking system call, which gets
canceled by the occurrence of a signal. We used 'nanosleep' for this purpose.
Once the current owner of the lock releases the lock, it sends a signal to the
next applicant of the lock. Because signals are buffered by the kernel, they
are guaranteed to be received by the targeted thread. However, in situations
with much lock contention, we observed the case where the signal was delivered
just before the to-be-blocked thread could enter the 'nanosleep' syscall. In
this case, the signal was not delivered at the next entrance into the kernel
(when entering 'nanosleep') but earlier. Even though the signal handler was
invoked, we found no elegant way to handle the signal such that the subsequent
'nanosleep' call would get skipped. So we decided to leave our signal-based
solution behind and went for the mainstream 'futex' mechanism instead.
Using this mechanism required us to re-design the internal lock API, which was
originally designed with the notion of thread IDs. The 'Native_thread_id' type,
which was previously used in the lock-internal 'Applicant' class to identify a
thread to be woken up, was not suitable anymore for implementing this change.
Hence, we replaced it with the 'Thread_base*' type, which also has the positive
effect of making the public 'base/cancelable_lock.h' header file
platform-independent.
In addition to reworking the basic locking primitives, we changed the
'Object_pool' data structure to become safer to use. The former 'obj_by_*'
functions have been replaced by 'lookup_and_lock' that looks up an object and
locks it in one atomic operation. Additionally, the case that an object may
already be in destruction is handled gracefully. In this case, the lookup will
return that the object is not available anymore.
Low-level OS infrastructure
###########################
Notification mechanism for the file-system interface
====================================================
To support dynamic system scenarios, we extended Genode's file-system interface
with the ability to monitor changes of files or directories, similar to the
inotify mechanism on Linux but simpler. The new 'File_system::sigh' function
can be used to install a signal handler for an open file node. When a node is
closed after a write operation, a prior registered signal handler for this file
gets notified. Signal handlers can also be installed for directories. In this
case, the signal handler gets informed about changes of immediate nodes hosted
in the directory, i.e., the addition, renaming, or removal of nodes.
The 'ram_fs' server has been enhanced to support the new interface. So any file
or directory change can now be observed by 'ram_fs' clients.
New adapter from file-system to ROM session interface
=====================================================
The new 'fs_rom' server translates the 'File_system' session interface to the
'ROM' session' interface. Each request for a ROM file is handled by looking
up an equally named file on the file system. If no such file can be found,
then the server will monitor the file system for the creation of the
corresponding file. Furthermore, the server reflects file changes as signals
to the ROM session.
There currently exist two limitations: First, symbolic links are not handled.
Second, the server needs to allocate RAM for each requested file. The RAM is
always allocated from the RAM session of the server. Thereby, the RAM quota
consumed by the server depends on the client requests and the size of the
requested files. Therefore, one instance of the server should not be used by
untrusted clients and trusted clients at the same time. In such situations,
multiple instances of the server could be used.
The most interesting feature of the 'fs_rom' server is the propagation of
file-system changes as ROM module changes. This clears the way to use this
service to supply dynamic configurations to Genode programs.
Dynamic re-configuration of the init process
============================================
The init process has become able to respond to configuration changes by
restarting the scenario using the new configuration. To make this feature
useful in practice, init must not fail under any circumstances. Even on
conditions that were considered previously as fatal and led to the abort of
init (such as ambiguous names of the children or misconfiguration in general),
init must stay alive and responsive to configuration changes.
With this change, the init process is one of the first use cases of the dynamic
configuration feature enabled via the 'fs_rom' service and the new file-system
notifications. By supplying the configuration of an init instance via the
'fs_rom' and 'ram_fs' services, the configuration of this instance gets fetched
from a file of the 'ram_fs' service. Each time, this file is changed, for
example via VIM running within a Noux runtime environment, the init process
re-evaluates its configuration.
In addition to the support for dynamic re-configurations, we simplified the use
of conditional session routing, namely the '<if-args>' mechanism. When matching
the 'label' session argument using '<if-args>' in a routing table, we can omit
the child name prefix because it is always the same for all sessions
originating from the child anyway. By handling the matching of session labels
as a special case, the expression of label-specific routing
becomes more intuitive.
Timer interface turned into asynchronous mode of operation
==========================================================
The 'msleep' function of 'Timer::Session' interface is one of the last relics
of blocking RPC interfaces present in Genode. As we try to part away from
blocking RPC calls inside servers and as a means to unify the timer
implementation across the many different platforms supported by Genode, we
changed the interface to an asynchronous mode of operation.
Synchronous blocking RPC interfaces turned out to be constant sources of
trouble and code complexity. E.g., a timer client that also wants to respond to
non-timer events was forced to be a multi-threaded process. Now, the blocking
'msleep' call has been replaced by a mechanism for programming timeouts and
receiving wakeup signals in an asynchronous fashion. Thereby signals
originating from the timer can be handled, along with signals from other signal
sources, by a single thread. Once a timer client has registered a signal
handler using the 'Timer::sigh' function, it can program timeouts using the
functions 'trigger_once' and 'trigger_periodic', which take an amount of
microseconds as argument. For maintaining compatibility and convenience, the
interface still contains the virtual 'msleep' function. However, it is not an
RPC function anymore but a mere client-side wrapper around the 'sigh' and
'trigger_once' functions. For use cases where sleeping at the granularity of
milliseconds is too coarse (such as udelay calls by device drivers), we added
a new 'usleep' call, which takes a number of microseconds as argument.
As a nice side effect of the interface changes, the platform-specific
implementations could be vastly unified. On NOVA and Fiasco.OC, the need to use
one thread per client has vanished. As a further simplification, we changed the
timer to use the build system's library-selection mechanism instead of
providing many timer targets with different 'REQUIRES' declarations. This
reduces the noise of the build system. For all platforms, the target at
'os/src/drivers/timer' is built. The target, in turn, depends on a 'timer'
library, which is platform-specific. The various library description files are
located under 'os/lib/mk/<platform>'. The common bits are contained in
'os/lib/mk/timer.inc'.
Since the 'msleep' call is still available from the client's perspective,
the change of the timer interface does not imply an API incompatibility.
However, it provides the opportunity to simplify clients in cases that required
the maintenance of a separate thread for the sole purpose of
periodic signal generation.
Loader
======
The loader is a service that enables its clients to dynamically create Genode
subsystems. Leveraging the new fault-detection support described in section
[New fault-detection mechanism], we enabled loader clients to respond to
failures that occur inside the spawned subsystem. This is useful for scenarios
where subsystems should be automatically restarted or in situations where the
system should enter a designated failsafe mode once an unexpected fault
happens.
The loader provides this feature by installing an optional client-provided
fault handler as default CPU exception handler and a RM fault handler for all
CPU and RM sessions of the loaded subsystem. This way, the failure of any
process within the subsystem gets reflected to the loader client as a signal.
The new 'os/run/failsafe.run' test illustrate this mechanism. It covers two
cases related to the loader, which are faults produced by the immediate child
of the loader and faults produced by indirect children.
Focus events for the nitpicker GUI server
=========================================
To enable a way for applications to provide visual feedback to changed keyboard
focus, we added a new 'FOCUS' event type to the 'Input::Event' structure. To
encode whether the focus was entered or left, the former 'keycode' member is
used (value 0 for leaving, value 1 for entering). Because 'keycode' is
misleading in this context, the former 'Input::Event::keycode' function was
renamed to 'Input::Event::code'. The nitpicker GUI server has been adapted to
deliver focus events to its clients.
NIC bridge with support for static IP configuration
===================================================
NIC bridge is a service that presents one physical network adaptor as many
virtual network adaptors to its clients. Up to now, it required each client
to obtain an IP address from a DHCP server at the physical network. However,
there are situations where the use of static IPs for virtual NICs is useful.
For example, when using the NIC bridge to create a virtual network between
the lighttpd web server and the Arora web browser, both running as Genode
processes without real network connectivity.
The static IP can be configured per client of the NIC bridge using a '<policy>'
node of the configuration. For example, the following policy assigns a static
address to a client with the session label "lighttpd".
!<start name="nic_bridge">
! ...
! <config>
! <policy label="lighttpd" ip_addr="10.0.2.55"/>
! </config>
!</start>
Of course, the client needs to configure its TCP/IP stack to use the assigned
IP address. This can be done via configuration arguments examined by the
'lwip_nic_dhcp' libc plugin. For the given example, the configuration for the
lighttpd process would look as follows.
!<start name="lighttpd">
! <config>
! <interface ip_addr="10.0.2.55"
! netmask="255.255.255.0"
! gateway="10.0.2.1"/>
! </config>
!</start>
Libraries and applications
##########################
New terminal multiplexer
========================
The new 'terminal_mux' server located at 'gems/src/server/terminal_mux' is able
to provide multiple terminal sessions over one terminal-client session. The
user can switch between the different sessions using a keyboard shortcut, which
brings up an ncurses-based menu.
The terminal sessions provided by terminal_mux implement (a subset of) the
Linux terminal capabilities. By implementing those capabilities, the server
is interchangeable with the graphical terminal ('gems/src/server/terminal').
The terminal session used by the server is expected to by VT102 compliant.
This way, terminal_mux can be connected via an UART driver with terminal
programs such as minicom, which typically implement VT102 rather than the Linux
terminal capabilities.
When started, terminal_mux displays a menu with a list of currently present
terminal sessions. The first line presents status information, in particular
the label of the currently visible session. A terminal session can be selected
by using the cursor keys and pressing return. Once selected, the user is able
to interact with the corresponding terminal session. Returning to the menu is
possible at any time by pressing control-x.
For trying out the new terminal_mux component, the 'gems/run/termina_mux.run'
script sets up a system with three terminal sessions, two instances of Noux
executing VIM and a terminal_log service that shows the log output of both Noux
instances.
New ported 3rd-party libraries
==============================
To support our forthcoming port of Git to the Noux runtime environment, we
have made the following libraries available via the libports repository:
* libssh-0.5.4
* curl-7.29.0 (for now the port is x86_* only because it depends on libcrypto,
which is currently not tested on ARM)
* iconv-1.14
Device drivers
##############
Besides the changes concerning the use of IOMMUs, the following device driver
have received improvements:
:UART drivers:
The OMAP4 platform support has been extended by a new UART driver, which
enables the use of up to 4 UART interfaces. The new driver is located at
'os/src/drivers/uart/omap4'.
All UART drivers implement the 'Terminal::Session' interface, which
provides read/write functionality accompanied by a function to determine
the terminal size. The generic UART driver code shared among the various
implementations has been enhanced to support the detection of the terminal
size using a protocol of escape sequences. This feature can be enabled by
including the attribute 'detect_size="yes"' in the policy of a UART client.
This is useful for combining UART drivers with the new 'terminal_mux'
server.
:ACPI support for 64-bit machines:
In addition to IOMMU-related modifications, the ACPI driver has been enhanced
to support 64-bit machines and MCFG table parsing has been added.
:PCI support for IOMMUs:
With the added support of IOMMUs, the 'Pci::Session' interface has been
complemented with a way to obtain the extended PCI configuration space in the
form of a 'Genode::Dataspace'. Also, the interface provides a way to allocate
DMA buffers for a given PCI device. Device drivers that are meant to be used
on system with and without IOMMUs should use this interface rather than
core's RAM session interface to allocate DMA buffers.
:Real-time clock on x86:
Up to now, the x86 real-time clock driver served as a mere example for
accessing I/O ports on x86 machines but the driver did not expose any service
interface. With the newly added 'os/include/rtc_session' interface and the
added support of this interface in the RTC driver, Genode programs have now
become able to read the real-time clock. Currently, the interface is used by
the Vancouver VMM.
:USB driver restructured, support for Arndale board added:
While adding support for the Exynos-5-based Arndale board, we took the
chance to restructure the driver to improve portability to new
platforms. The most part of the driver has become a library, which is
built in a platform-specific way. The build system automatically selects
the library that fits for the platform as set up for the build directory.
Platforms
#########
NOVA
====
The NOVA base platform received major improvements that address the kernel
as well as Genode's NOVA-specific code. We pursued two goals with this line
of work. The first goal was the use of NOVA in highly dynamic settings, which
was not possible before, mainly due to lacking kernel features. The second
goal was the use of IOMMUs.
NOVA is ultimately designed for accommodating dynamic workloads on top of the
kernel. But we found that the implementation of crucial functionality was
missing. In particular, the kernel lacked the ability to destroy all kinds of
kernel objects and to reuse memory of kernel objects that had been destroyed.
Consequently, when successively creating and destroying kernel objects such as
threads and protection domains, the kernel would eventually run out of memory.
This issue became a show stopper for running the Genode tool chain on NOVA
because this scenario spawns and destroys hundreds of processes. For this
reason, we complemented the kernel with the missing functionality. This step
involved substantial changes in the kernel code. So our approach of using the
upstream kernel and applying a hand full of custom patches started to show its
limitations.
To streamline our work flow and to track the upstream kernel in a structured
way, we decided to fork NOVA's Git repository and maintain our patches in our
fork. For each upstream kernel revision that involves kernel ABI changes, we
create a separate branch called "r<number>". This branch corresponds to the
upstream kernel with our series of custom patches applied (actually rebased) on
top. This way, our additions to the upstream kernel are well documented. The
'make prepare' mechanism in the base-nova repository automates the task of
checking out the right branch. So from the Genode user's point of view, this
change is transparent.
The highly dynamic application scenarios executed on NOVA triggered several
synchronization issues in Genode's core process that had not been present on
other base platforms. The reason for those issues to occur specifically on NOVA
lies in the concurrent page fault handling as employed on this base platform.
For all classical L4-like kernels and Fiasco.OC, we use one global pager thread
to resolve all page faults that occur in the whole Genode system. In contrast,
on NOVA we use one pager thread per user thread. Consequently, proper
fine-grained synchronization between those pager threads and the other parts of
core is mandated. Even though the immediate beneficiary of these changes is the
NOVA platform, many of the improvements refer to generic code. This paves the
ground for scaling the page-fault handling on other base platforms (such as
Fiasco.OC) to multiple threads. With these improvements in place, we are able
to successfully execute the 'noux_tool_chain_nova' scenario on the NOVA kernel
and build Genode's core on NOVA. That said, however, not all issues are covered
yet. So there is still a way left to go to turn base-nova into a base platform
that is suitable for highly dynamic scenarios.
The second goal was the use of NOVA's IOMMU support on Genode. This topic is
covered in detail in section [DMA protection via IOMMU].
To be able to use and debug Genode on NOVA on modern machines that lack legacy
comports, we either use UART PCI cards or the Intel's Active Management
Technology (AMT) mechanism. In both cases, the I/O ports to access the serial
interfaces differ from the legacy comports. To avoid the need for adjusting the
I/O port base addresses per platform, we started using the chain-boot-loader
called "bender" developed by the Operating Systems Group of TU Dresden,
Germany. This boot loader is started prior the kernel, searches the PCI bus for
the first suitable device and registers the corresponding I/O port base address
at the bios data area (BDA). Genode's core, in turn, picks the I/O port base
address up from the BDA and uses the registered i8250 serial controller for its
LOG service.
Execution on bare hardware (base-hw)
====================================
The base-hw platform enables the use of Genode on ARM-based hardware without
the need for a 3rd-party kernel.
With the new release, the range of supported ARM-based hardware has been
extended to cover the following additional platforms. With the previous
release, we introduced the support for Freescale i.MX family of SoC, starting
with i.MX31. The current release adds support for the i.MX53 SoC and adds
a user-level timer driver for this platform. With the Samsung Exynos 5, the
first Cortex-A15-based SoC has entered the list of supported SoCs. Thanks to
this addition, Genode has become able to run on the
[http://www.arndaleboard.org - Howchip Arndale board]. At the current state,
core and multiple instances of init can be executed but drivers for peripherals
are largely missing. Those will be covered by our ongoing work with this SoC.
The added platforms are readily available via the 'create_builddir' tool.
To make base-hw practically usable on real hardware (i.e., the Pandaboard),
support for caches has been implemented. Furthermore, the implementation of the
signalling API underwent a redesign, which leverage the opportunities that
arise with tailoring a kernel specifically to the Genode API. As a side-benefit
of this endeavour, we could unify the 'base/signal.h' header with the generic
version and thereby took another step towards the unification of the Genode
headers across different kernels.
Microblaze platform removed
===========================
The 'base-mb' platform has been removed because it is no longer maintained.
This platform enabled Genode to run directly on the Xilinx Microblaze softcore
CPU. For supporting the Microblaze CPU architecture in the future, we might
consider integrating support for this architecture into base-hw. Currently
though, there does not seem to be any demand for it.
Fiasco.OC forked, support for Exynos 5 SoC added
================================================
In the last release cycle, we went beyond just using the Fiasco.OC kernel and
started to engage with the kernel code more intensively. To avoid that the
management of a growing number of kernel patches goes out of hand, we forked
the Fiasco.OC kernel and conduct our development in our Fiasco.OC Git
repository. When using the 'make prepare' mechanism in the 'base-foc'
repository, the new Git repository will be used automatically. There exists a
dedicated branch for each upstream SVN revision that we use. We started with
updating Fiasco.OC to the current revision 47. Hence, the current branch used
by Genode is named "r47". The branch contains the unmodified state of the
upstream SVN repository with our modifications appearing as individual commits
on top. This makes it easy to keep track of the Genode-specific modifications.
Please note that the update to Fiasco.OC requires minor adaptations inside
the 'ports-foc' repository. So for using L4Linux, "make prepare" must be
issued in both repositories 'base-foc' and 'ports-foc'.
Speaking of engaging with the kernel code, the most profound improvement is
the support for the Samsung Exynos-5-based Arndale board that we added to the
kernel. This goes hand in hand with the addition of this platform to Genode.
For creating a build directory targeting the Arndale board, just specify
"foc_arndale" to the 'create_builddir' tool. At the time of the release,
several basic scenarios including the timer driver and the USB driver are
working. Also, both Cortex-A15 CPUs of the Exynos 5 SoC are operational.
However, drivers for most of the peripherals of the Exynos-5 SoC are missing,
which limits the current scope of Genode on this platform.
Linux
=====
Since the base-linux platform became used for more than a mere development
vehicle, we are revisiting several aspects of this base platform. In the last
release, we changed the synchronous inter-process-communication mechanism to
the use of SCM rights. For the current release, it was time to have a closer
look at the memory management within core. The Linux version of core used a
part of the BSS to simulate access to physical memory. All dataspaces would
refer to a portion of 'some_mem'. So each time when core would access the
dataspace contents, it would access its local BSS. For all processes outside of
core, dataspaces were represented as files. We have now removed the distinction
between core and non-core processes. Now, core uses the same 'Rm_session_mmap'
implementation as regular processes. This way, the 'some_mem' could be
abandoned. We still use a BSS variable for allocating core-local meta data
though. The major benefit of this change is the removal of the artificial
quota restriction that was imposed by the predefined size of the 'some_mem'
array. Now, the Linux base platform can use as much memory as it likes. Because
the Linux kernel implements virtual memory, we are not bound by the physical
memory. Hence, the available quota assigned to the init process is almost
without bounds.
To implement the fault-detection mechanism described in section
[New fault-detection mechanism] on Linux, we let core catch SIGCHLD signals of
all Genode processes. If such a signal occurs, core determines the process that
produced the signal by using 'wait_pid', looks up the CPU session that belongs
to the process and delivers an exception signal to the registered exception
handler. This way, abnormal terminations of Genode processes are reflected to
the Genode API in a clean way and Genode processes become able to respond to
terminating Genode child processes.
OKL4
====
The audio stub driver has been removed from OKLinux. Because of the changed
'Audio_out::Session' interface, we needed to decide on whether to adapt the
OKLinux stub driver to the changed interface or to remove the stub driver.
Given the fact that OKLinux is not actively used, we decided for the latter.

943
doc/release_notes-13-05.txt
View File

@@ -1,943 +0,0 @@
===============================================
Release notes for the Genode OS Framework 13.05
===============================================
Genode Labs
With Genode 13.05, we have diverged quite a bit from the feature-laden plans
laid out in our [http://genode.org/about/road-map road map] as we realized
that consolidating and optimizing the current feature set will have a more
sustainable effect than functional enhancements at this point. In particular,
we addressed the problem that the ever growing diversity of platforms imposes
on the quality and coverage of testing. We also desired to extend our
systematic testing efforts to real hardware platforms, and to have a mechanism
for detecting performance regressions. Section
[Automated quality-assurance testing] details how we approached these
challenges, and how we went on analyzing Genode's network performance in
particular.
That said, we haven't completely restrained ourself from implementing new
features. Closely related to test automation but very useful in other
situations, we improved the terminal infrastructure in order to enable the
interactive use of dynamic system scenarios in headless situations. Section
[Terminal infrastructure] introduces a new command-line interface for managing
Genode subsystems.
With regard to platform support, the current release follows up on the
hardware support added in the previous releases. For Samsung Exynos-5-based
platforms, drivers for USB-3, fast-ethernet networking, gigabit networking,
eMMC, and SATA have been added. For Freescale i.MX53-based devices, new
drivers for display, touchscreen, and GPIO have become available. The
OMAP4 display driver has been enhanced to cover both LCD displays and HDMI.
Our custom base-hw kernel has been enabled on the Raspberry Pi
board. Finally, Linux/ARM was added to accompany Linux/x86 as a fully usable
Genode base platform.
Automated quality-assurance testing
###################################
One of the greatest challenges of the Genode OS Framework is preventing
regressions in the face of the growing number of supported platforms.
The challenge stems from the fact that the space of Genode scenarios grow
two-dimensional. On one axis, the software stack on top of Genode gets more
and more complex, which calls for contiguous testing. On the other axis, there
is a growing number of kernel and hardware platforms to support.
In principle, there are even more dimensions, for example the diversity
of tool chains or the diversity of the OS used on the development machine.
Luckily, the problem of tool-chain diversity could be mitigated with the
introduction of the Genode tool chain since version 11.11, which was a huge
relief. However, the mentioned two dimensions cannot be avoided. Because
manual testing of manifold scenarios of component compositions on top of many
different kernels became infeasible, we automated the task of building and
testing years ago.
The automated builder checks out the staging branch of Genode, prepares
the repositories that integrate 3rd-party code, and builds the software
for 12 different kernel/platform combinations. Not all 3rd-party software
packages are built for each combination though. But we make sure that each
piece of software is exposed to different combinations of CPU architectures
and kernels.
The build test is accompanied with automated runtime tests of various
run scripts on Qemu. Each run script listed in 'tool/autopilot.lst' is
executed on each kernel using the autopilot tool. The tests range from
stimulating low-level mechanisms (such as signal, timer, and ldso) to complex
scenarios (such as testing networking with L4Linux, or running Noux).
Both build and runtime tests are executed daily. If any of the
tests fail, the Genode developers receive a notification email.
Once all tests are passed, the staging branch can be merged into the master
branch. This way, we spare the users of Genode to deal with intermediate
problems introduced in the staging branch.
The build and runtime tests have become a fundamental tool for our
development work. With the growing variety of real hardware
(as opposed to hardware emulated via Qemu), however, our existing solution
was falling short. Even though our tests confirm that Genode is running
happily on Qemu, they won't help us to detect regressions in our device
drivers for non-Qemu hardware such as Pandaboard, Arndale, or modern PC
hardware. Furthermore, we are increasingly focussing on performance
considerations. In order to be a viable OS platform, Genode does not only need
to be able to do networking, but networking performance must be on par with
mainstream OSes. This raises the new challenge to extend our
continuous-testing tools to become continuous-benchmarking tools. The ultimate
goal is to monitor the performance of Genode on real hardware over long
periods of development.
In this release cycle, we attacked this problem in two steps. First, we
enabled Genode's run tool to target not only Qemu but real hardware, with the
premise that existing run scripts must not be changed. The second step is the
creation of new run scripts that perform benchmarks in an automated fashion.
By aggregating the results of this automatically executed benchmarks, we can
correlate performance effects with commits in our code repository.
Targeting real hardware via the run tool
========================================
In the following, we briefly describe the procedure to execute run scripts
on native hardware, for both Intel-based x86 machines and ARM-based platforms.
TFTP boot x86
~~~~~~~~~~~~~
The following description uses NOVA as an example to illustrate the usage.
Other base platforms are supported as well and can be configured analogously.
[http://os.inf.tu-dresden.de/~us15/pulsar/ - Pulsar] is a tiny boot loader
that uses PXE to fetch boot images via TFTP over the network. On the x86
architecture, Genode supports the automatic generation of Pulsar configuration
files, which can be placed directly onto a TFTP server. Genode can be booted
via Pulsar using the following steps:
* On the x86 test machine, enable "PXE boot feature" in the BIOS.
* When booting, the machine will look for a DHCP server announcing a TFTP server.
So you need to make sure to have both the DHCP server and the TFTP server
configured such that the 'pulsar' binary will be loaded as PXE binary.
* After the PXE BIOS of the test machine has loaded and started the pulsar
binary, Pulsar will look on the TFTP server for a file called
'config-XX-XX-XX-XX-XX-XX', where the sequence of 'XX' corresponds to the
MAC address of the test machine.
For example, if the MAC of the network card is 01:02:03:04:05:06, Pulsar
would request a file called 'config-01-02-03-04-05-06'.
* Using this configuration file, we direct Pulsar to the configuration
generated by the run tool. I.e., it should look as follows
! root /tftpboot/nova
! config config-00-00-00-00-00-00
The lines above tell pulsar to load another config file, which contains the
actual configuration. To instruct the run script to actually generate the
'config-00-00-00-00-00-00' file, set the following environment variables in
your shell prior executing the run script:
! export PXE_TFTP_DIR_BASE=/tftpboot
! export PXE_TFTP_DIR_OFFSET=/nova
The two-staged configuration of Pulsar may look overly complicated at first
sight but has the benefit that the run tool does not need to know the MAC
address of the test machine in order to generate the Pulsar configuration
file.
* Create a symbolic link '/tftpboot/nova' pointing to the corresponding
Genode build directory.
* The next time 'make run/printf' is invoked,
the run script will generate the 'config-00-00-00-00-00-00' in
'/tftpboot/nova'.
* When rebooting the test machine, it will load and start the printf test.
TFTP boot using U-Boot
~~~~~~~~~~~~~~~~~~~~~~
Configure your U-Boot boot loader to load the images via TFTP.
The remainder of the procedure is similar to the description for x86 above.
On ARM platforms, the run tool automatically generates the uBoot image and
creates a symbolic link into the TFTP directory.
* Pandaboard:
! export PXE_TFTP_DIR_BASE=/tftpboot
! export PXE_TFTP_DIR_OFFSET=/panda
! ln -s <genode-build-dir> /tftpboot/panda
! RUN_OPT="--target uboot" make run/printf
* Arndale board:
! export PXE_TFTP_DIR_BASE=/tftpboot
! export PXE_TFTP_DIR_OFFSET=/arndale
! ln -s <genode-build-dir> /tftpboot/panda
! RUN_OPT="--target uboot" make run/printf
Output and reset with Intel's AMT
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Most modern x86-based machines lack a COM port, which is normally used for
kernel debug messages as well as LOG messages printed by Genode's core.
However, Intel's Advanced Management Technology (AMT) can be used to obtain
the serial output of the test machine and to reset the test machine. To use
AMT with Genode's run tool, install the 'amtterm' package (version 1.3 is
known to work well) and set the following environment variables, specifying
the IP address of the test machine and the AMT password.
! export AMT_TEST_MACHINE_IP=XXX.XXX.XXX.XXX
! export AMT_TEST_MACHINE_PWD=XXXXXXXXX
Via setting the RUN_OPT environment variable, we instruct the run tool to use
AMT instead of Qemu. The following command will reset the test machine, the test
machine will load the binaries of the printf run script via PXE, and we will be
able to see the serial output of the test machine through Intel's AMT Serial
Over Line (SOL),
! RUN_OPT="--target amt" make run/printf
Output via a COM port (UART)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
If the x86 test machine, Pandaboard or Arndale test board is connected
via UART, the run tool can use a specified command to interact with it.
For example, if the UART interface of the test machine is connected directly
to the host machine at /dev/ttyUSB3, and the picocom tool is available,
the following command can be used to establish a connection:
! RUN_OPT="--target serial --serial-cmd \"picocom -b 115200 /dev/ttyUSB3\"" make run/printf
Alternatively, if the board is connected to some remote machine, which exports
the corresponding serial line via TCP/IP, the socat tool can be used for
communicating with the remote test machine:
! RUN_OPT="--target serial --serial-cmd \"socat - tcp:10.0.0.1:2000\"" make run/printf
Reset via a IP power plug NETIO-230B from Koukaam
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
At Genode Labs, we use a NETIO-230B power plug to automate power-cycling ARM
boards. This power plug can be controlled over the network. For example, if
the Pandaboard is connected to power port 3, the following command will
automatically turn on the board when the run script is started:
! RUN_OPT="--target uboot --target reset --reset-port 2 --reset-ip 10.0.0.1 --reset-user admin --reset-passwd secret" make run/printf
The '--target reset' option can be combined with '--target uboot' to
instruct the run tool to boot via TFTP (as described above) and take care
of power cycling. When the run script has finished, the specified port
will be automatically switched off by the run tool.
Of course, the IP address settings, as well as the actual user name and
password, to access the NETIO-230B power plug, have to be adjusted accordingly.
Automated benchmarking
======================
With the '--target' features added to the run tool, the road is paved to
obtain benchmark results in an automated fashion. Currently, we are most
interested in exploring the network-performance characteristics of Genode.
Network performance can be explored at different levels. We started with
looking at raw driver performance, then looked at the overhead of separating
the network application from the device driver (and thereby introducing
inter-process communication overhead), and finally explored the effects
of the TCP/IP stack.
For pursuing the packet-level performance measurements, we crafted a library
called 'net-stat', which contains the application logic of a low-level
benchmark operating at network-packet level. This library has been
successively incorporated into the 'dde_ipxe' NIC driver and the 'usb_drv'
(NIC driver via ethernet-over-USB) to measure the raw driver performance
without any microkernel overhead or TCP/IP protocol overhead.
To see the influence of the inter-process communication, namely the
packet-stream interface employed by Genode's NIC-session interface,
we implanted the same net-stat library into a NIC-session client. This
experiment enables us to compare the operation of the NIC driver
with the operation of a NIC driver separated from the NIC
application.
The raw networking tests can be executed automatically using the set
of 'network_test_nic*.run' scripts located at 'os/run'.
The scenario sends raw ethernet packets from the host machine to the
target machine. Three tests are provided:
The 'network_test_nic_raw.run' test measures the net-stat-instrumented driver
(of usb_drv and net_drv respectively) to observe the raw receive performance.
The 'network_test_nic_raw_client.run' test implements the benchmark in a
NIC-session client connected to the NIC driver running as a separate
component whereas the NIC driver is not instrumented.
The 'network_test_nic_raw_bridge_client.run' test further adds a NIC bridge
in-between the driver and the NIC-session client.
In addition to analyzing the performance on a low level, we investigated
the effects of TCP/IP for the application performance. This topic is
covered in more detail in Section [TCP/IP performance].
Terminal infrastructure
#######################
Closely related to the quality-assurance measures detailed in the previous
section, there is the arising need to interact with increasingly complex system
scenarios in headless settings. In particular when executing tests remotely on a
development board, manual user-interaction via a GUI
becomes impractical. We vastly prefer a low-bandwidth textual interface
in such situations. But how should a textual user interface for dynamic
systems comprised of many components look like? This is particularly difficult
because most development boards are equipped with merely a single UART
connector.
On a normal Genode system, the UART connector is typically used
by the kernel debugger to print debugging output, or for the interactive
use of a debugger. This leaves no interface for interacting with Genode
components. So how can we expose complex scenarios, such as concurrently
running several instances of Genode subsystems, to the user?
Our solution consists of three parts: A pseudo UART driver for Genode
that uses the kernel debugger as back end, a terminal-multiplexing
facility running on the reference platform, and a command-line based
tool for interacting with Genode. By combining those, the user
can interact with the kernel debugger, a Genode command line, and the
consoles of executed Linux instances over a single serial connection.
The pseudo UART driver called kdb_uart_drv is a Genode service that
implements the 'Uart::Session' interface. Therefore, it can be combined
with all components that use the 'Uart::Session' or the 'Terminal::Session'
interfaces, for example the Noux runtime environment, the terminal_log
service (for displaying LOG messages via the terminal interface), L4Linux, or
programs linked against the 'libc_terminal' plugin. The kdb_uart_drv component
is located at 'os/src/drivers/uart/kdb'. It does not access a real UART device
but rather uses the user-level bindings of the kernel debugger to indirectly
read and write data over the UART interface.
[image kdb_uart_drv 65%]
The kdb_uart_drv driver used for sharing one UART among the kernel
debugger, core's LOG service, and a terminal client application
running on Genode.
Figure [kdb_uart_drv] illustrates the relationship between the kernel
debugger, core's LOG service, and kdb_uart_drv. Because write operations
target the kernel debugger directly, core's LOG service gets bypassed. Output
written to the kdb_uart_drv will directly appear at the terminal program of
the host system. Because kdb_uart_drv has
direct access to the host terminal, it can leverage all facilities of the host
terminal, in particular various escape sequences for terminal manipulations.
For reading from the kernel debugger, there is no way to block for UART input.
Hence, the kdb_uart_drv periodically polls for new input with a period of 20
milliseconds. If new input is available, the driver reads as many characters as
available at once. So the runtime overhead of polling is negligible. To test
kdb_uart_drv as individual component, there is a run script provided at
'os/run/kdb_uart_drv.run'.
Thanks to kdb_uart_drv, both the kernel debugger and Genode can
share one single UART connection. So we have a principal way to let the user
interact with a Genode component that uses the 'Terminal::Session' interface.
However, typical system scenarios should accommodate not just a single program
but multiple Linux instances and native Genode applications simultaneously,
each requiring a dedicated 'Terminal::Session'. Hence, we need a way to
multiplex the 'Terminal::Session' interface between those clients. Our
multiplexing solution comes in the form of a component called terminal_mux,
which we just introduced in the
[http://genode.org/documentation/release-notes/13.02#New_terminal_multiplexer - previous release].
It uses a single terminal connection to implement a text-based user interface
to multiple virtual terminal consoles.
[image terminal_mux 40%]
Operation of the terminal_mux service.
Figure [terminal_mux] depicts the basic functioning of this component. For
terminal_mux clients, the service implements the Linux terminal capabilities.
For doing that, it shares large parts of the implementation of the existing
Genode terminal program. For each client, terminal_mux renders the client
output into a client-specific text-screen buffer. So any number of clients can
perform output on terminal_mux concurrently. According to the selection by
the user, terminal_mux periodically translates one client buffer (the
foreground buffer) to escape sequences as understood by the host terminal. This
translation is performed using the ncurses library. The user can pick the
foreground buffer using an interactive menu that can be activated via the
keyboard shortcut _Control-x_.
By combining kdb_uart_drv with terminal_mux, we created a flexible way
to let the user interact with many Genode applications. The last part
missing for a real dynamic system is a text-based command interface to
start and stop Genode subsystems. This functionality is provided by the
new cli_monitor component located at 'os/src/app/cli_monitor'.
It uses the 'Terminal::Session' interface to present a simple interactive
command line with commands for starting and stopping Genode subsystems,
entering the kernel debugger, and showing status information. It provides
tab completion and inline help to make it easily explorable. The cli_monitor
component is integrated in the scenario of the 'terminal_mux.run' script
mentioned above. Because cli_command is a 'Terminal::Session' client, it can
be interfaced with terminal_mux. This composition is illustrated by Figure
[uart_overview].
[image uart_overview 100%]
Overview of the terminal infrastructure as employed in the
demonstration scenario.
Note that in some situations, e.g., when killing subsystems, the kernel, core,
or the init process may print LOG messages. Because those messages are
naturally not routed through terminal_log, they will interfere with the
operation of terminal_mux and thereby result in visible inconsistencies.
Pressing _Control-x_ will clear such artifacts. This will bring up the
terminal_mux menu, which implicitly triggers the redraw of the entire terminal.
Base framework
##############
The current release comes with incremental improvements of the MMIO framework
API and a new utility to ease the synchronized accesses to otherwise
unsynchronized class interfaces.
:MMIO framework improvements:
For native Genode device drivers, we consistently use our
[http://genode.org/documentation/release-notes/12.02#MMIO_access_framework - MMIO framework API].
These utilities help us to safeguard the access to individual bit fields of
memory-mapped device registers and cleanly separate the declaration of device
registers from the driver logic. During the increased use of the API, we
observe that the 'Genode::Mmio' class template operates mostly on addresses
that belong to dataspaces provided by core's IO_MEM service. Those dataspaces
are typically obtained via the 'Attached_io_mem_dataspace' convenience class,
which requests the dataspace and attaches it to the local address
space at once. To further reduce repetitive code, we introduced the new
'Attached_mmio' class (located at 'os/attached_mmio.h'), which handles the
common case of making the content of a IO_MEM dataspace available through
register definitions using the 'Mmio' utility. Furthermore, the MMIO framework
API has been enhanced with a variant of the 'Mmio::wait_for()' function that
waits for whole register values rather than bits.
:Synchronized interfaces:
Most Genode programs are multi-threaded, which makes the proper use of locks
inevitable. For most data structures, Genode does not implicitly manage the
locking but expects the user of the data structures to know what he is doing.
This way, we can avoid the locking overhead if a data structure is known to be
accessed by a single thread only. If accessed by multiple threads, we usually
wrap such data structures within an accessor interface that takes care of the
locking. For example, for the 'Allocator' interface, there exists a
corresponding 'Synchronized_allocator' interface wrapper. This technique works
well as long as the number of interfaces is low -- as is the case for Genode's
base API. However, as the wrapper code is for the most part pretty dumb, we'd
like to avoid it. Also, when using the Genode API to implement programs on
top, we do not anticipate manually creating such accessor wrappers. To ease
the creation of synchronized interfaces, we introduced the new
'Synced_interface' class template. It takes a pointer to an existing interface
and a lock as arguments. An instance of a 'Synced_interface' provides
synchronized access to the wrapped interface functions via the 'operator ()'.
Because the 'Synced_interface' does not provide any means to obtain the
unsynchronized version of the interface, once wrapped, the interface cannot be
misused by subsystems that get handed over a reference to a
'Synced_interface'. To see how to employ this utility, please have a look of
how we realize the synchronization within the Vancouver VMM (in particular,
the access to the motherboard).
Low-level OS infrastructure
###########################
TCP/IP performance
==================
On the course of the automated benchmarking described in Section
[Automated quality-assurance testing], we conducted the following steps
to enable benchmarks and to improve performance at the TCP/IP level.
At application level, we desire to compare our network performance with the
performance on GNU/Linux using commodity benchmarks. For this reason, netperf
has been ported to run as native Genode using the lwIP stack. This benchmark
allows us to systematically compare our results with those achieved by Linux.
The port of netperf is available in the ports repository.
In addition to running a commodity benchmark, we pursue synthetic benchmarks
that model the behaviour of typical application scenarios, for example, a
web server that receive many small requests. This is where the added
'test-ping_client' and 'test-ping_server' tests come into play. The test
is located at 'libports/src/test/lwip/pingpong'. It is used by the
series of 'network_test_*.run' scripts located at 'libports/run'. The
run scripts exercise the test in various scenarios and thereby allow us to
systematically explore the impact of the libc and NIC bridge on the
application performance.
# Using raw lwIP without the libc
# Like the first test, but with an instance of the NIC bridge in between the
test program and the driver.
# Using lwIP with the libc socket bindings
# Like the third test, but with NIC bridge added
To keep track of the lwIP development more closely, we switched to the
Git version of lwIP instead of using a source snapshot.
Furthermore, we incorporated "window scaling" support (RFC 1323) into our
version of lwIP as we identify the TCP window size as a limiting factor
of the TCP throughput achieved via lwIP.
C runtime
=========
We added support for "resolv" functionality to the libc_lwip_nic_dhcp plugin.
Normally, a file called 'resolv.conf' is expected to be located at '/etc'.
On Genode, however, we don't have a global file system, which makes this
way of configuration cumbersome. To ease the provision of a simple default
'resolv.conf' configuration, the plugin hands out the file as a virtual file.
The configuration automatically provides the DNS server address acquired by
lwIP via DHCP. If, for some reason, this policy is not desired, the feature
can be disabled via:
! <libc resolv="no" />
*Note that the configuration of the C runtime has changed*
To foster consistency of the libc configuration, we moved the static
network "interface" attributes into the 'libc' XML node. A new configuration
of static networking would look as follows:
! <libc ip_addr="..." netmask="..." gateway="..." />
Terminal
========
Genode's custom terminal implementation has been improved to better handle
widely used escape sequences.
The new version is able to handle two-argument SGR commands with
attribute/color arguments in any order, and supports the ED, EL0, and
CUB commands.
Because the terminal classes do not rely on any 3rd-party code, they
have been moved to the os repository at 'os/include/terminal'. This way,
we can use those classes by other components of the os repository such
as the new CLI monitor.
FS-LOG service
==============
Using the new FS-LOG service residing at 'libports/os/src/server/fs_log', log
messages of different processes can be redirected to files on a file-system
service. The assignment of processes to files can be expressed in the
configuration as follows:
! <start name="fs_log">
! <resource name="RAM" quantum="2M"/>
! <provides><service name="LOG"/></provides>
! <config>
! <policy label="noux" file="/noux.log" />
! <policy label="noux ->" file="/noux_process.log" />
! </config>
! </start>
In this example, all messages originating from the noux process are directed
to the file '/noux.log'. All messages originating from children of the noux
process end up in the file '/noux_process.log'.
Liquid FB
=========
Liquid FB is a virtual framebuffer service that uses the nitpicker GUI
server as back end. The virtual framebuffer is presented as a movable
window with a title bar. Until now, we used it primarily for demonstration
purposes, i.e., it is part of Genode's default demo scenario.
Thanks to our forthcoming adaptation of Qt5 to Genode, which requires a
very similar solution to interface Qt5's platform-abstraction layer (QPA) to
Genode, liquid FB got in the spotlight of this release.
First, we took the chance to update its configuration parameters to
become more consistent with similar services such as nit_fb. As liquid_fb was
originally conceived at a time when Genode's XML parser did not support
XML attributes, its configuration syntax used to be a bit arcane. This
has changed now. Apart from this cosmetic refinement, there are two prominent
new features: Support for resizing the framebuffer window with the
mouse and support for dynamic reconfiguration of the virtual framebuffer
via Genode's configuration mechanism.
When the liquid FB window gets resized by the user, the virtual framebuffer
emits a mode-changed signal to its client, which, in turn can handle the
event by re-acquiring the frame-buffer dataspace.
The added support for dynamic reconfiguration allows for changing the
properties of a liquid FB instance via Genode's configuration mechanism.
For example, the window position and size can be manipulated this way.
Furthermore, two new configuration options have been added. The
'resize_handle' option shows or hides the resize handle widget at the
lower-right window corner (by default, it is hidden). The 'decoration' option
defines whether window decorations should be visible (default is yes). Both
options can have the values "on" or "off".
3rd-party libraries
###################
The following 3rd-party libraries have been added or updated:
* To complement libSDL, we have added ports of SDL_ttf, SDL_image,
SDL_image, SDL_mixer, and SDL_loadso. Those additions to libSDL
are used by popular libSDL-based applications such as Tuxpaint.
They are now available at the libports repository.
* GNU FriBidi 0.19.5 added to the libports repository
* Qt4 updated to version 4.8.4
* zlib updated to version 1.2.8
Device drivers
##############
Unified driver names
====================
The growing diversity of supported hardware platforms calls for improved
conventions of how to name device drivers. Otherwise, run scripts that are
meant to support a wide range of platforms will eventually become more
and more complicated due to platform-dependent conditional configuration
snippets. For example, the default framebuffer drivers of the respective
platforms used to be called "vesa_drv" (for x86), "omap4_fb_drv", or "pl11x_drv".
In order to support the different platforms, run scripts that were otherwise
platform-agnostic had to explicitly deal with those differences.
To solve this issue, we introduced a generic SPEC values for device types, for
which a default driver is expected to exist. If a platform features a
framebuffer driver, it includes the SPEC value "framebuffer". On each
platform, the default driver for the respective device has the same name. So
each of "vesa_drv", "pl11x_drv", and "omap4_fb_drv" had been renamed to
"fb_drv". This is possible because the use of those drivers is mutually
exclusive.
The same convention has been applied to GPIO drivers as well. The
corresponding SPEC value is called "gpio". The driver binaries are called
"gpio_drv".
ATAPI
=====
LBA48 support has been added to the ATAPI driver. Thanks to Ivan Loskutov!
KDB UART driver for L4/Fiasco and Fiasco.OC
===========================================
The new KDB UART driver at 'os/src/drivera/uart/kdb' uses the kernel debugger
console as backend for input and output. This is useful in the case that only
one UART is available as described in Section [Terminal infrastructure].
Examples for using the kdb_uart_drv are available in the form of the run scripts
'ports-foc/run/l4linux.run' and 'os/run/kdb_uart_drv.run'.
Revised GPIO session interface
==============================
The original design of the GPIO session interface enabled the client of a
single session to interact with any number GPIO pins. Each function of the
interface took a GPIO number as first argument, which addressed the GPIO pin.
To simplify the interface and to enable fine-grained GPIO-assignment policies,
the interface has been changed to provide access to a single GPIO pin per
session only. At session creation time, the client specifies a single GPIO
pin, to which the session refers. This information can be evaluated for the
session routing. So access-control policies can be easily implemented per GPIO
pin. The server stores the pin as part of the session context and implicitly
uses the pin for operations on the session interface.
Furthermore, a generic driver interface for GPIO-class-device drivers
has been introduced. The new interface at 'os/include/gpio' alleviates the
need to implement the boilerplate code to interface the driver with Genode.
The existing GPIO drivers for OMAP4 and i.MX53 are the first beneficiaries of
these changes.
Exynos 5 SoC
============
After principally enabling the Exynos 5 SoC platform in the previous
release, we moved on with extending the device-driver coverage of this SoC. In
particular, we addressed USB networking, XHCI (USB-3), Gigabit networking over
USB-3, eMMC, and SATA.
The development of those device drivers follows our rationale that guided our
[http://genode.org/documentation/articles/pandaboard - previous work on the OMAP4 platform].
For the USB driver, we employed the device-driver-environment (DDE) approach
for reusing the Linux USB stack and the host controller drivers. In contrast,
the eMMC and SATA drivers are built as genuine Genode drivers with no
3rd-party code used.
Technically, the addition of Exynos-5 support to our USB driver was
an evolutionary step. It required us to add the corresponding EHCI
controller and to supply a few additions to the device-driver
environment. To simplify the driver, we decided to let the driver
rely on the platform initialization as performed by the U-Boot boot
loader. Since the initialization is performed during the boot process
already, there is no need to do this work twice. Because the platforms
supported by the USB driver become more and more diverse, we re-organized the
internal structure of the 'dde_linux' repository to keep those platforms well
separated. Furthermore, we reworked the memory management of the USB driver to
improve the utilization of the available RAM. The new solution employs Genode's
concept of managed dataspaces to manage a part of the local address-space
layout manually. This helps us to implement a fast translation of driver-local
virtual addresses to physical addresses as needed for issuing DMA requests.
The eMMC driver builds upon our protocol implementation for the SD-card
protocol, which was originally developed for the OMAP4 SD-card driver.
Because we kept the SD-card protocol implementation well separated
from the host-controller driver, it was possible to leverage parts of our
existing work for the eMMC driver. Because the eMMC protocol is an extension
of the SD-card protocol, however, we needed to enhance the protocol
implementation accordingly. The extension comprises support for the
MMC_SEND_EXT_CSD, MMC_SEND_OP_COND, and STOP_TRANSMISSION commands as well as
the MMC detection. The host controller driver was implemented from scratch
with the help of I/O access traces gathered from instrumenting the U-Boot boot
loader and the Linux kernel. The driver operates the eMMC in high-speed, 8-bit
mode at 52 MHz using DMA. The implementation can be found at
'os/src/drivers/sd_card/exynos5'.
The initial version of our new SATA driver for Exynos 5 has been implemented
from the ground up. Even though it is at an early stage, it has been
successfully tested with a UDMA-133 disk, e.g., our generic block test
is passed and the disk can be attached as a block device to an instance of
L4Linux.
Freescale i.MX SoC
==================
The support for the Freescale i.MX53 SoC has been extended by a number of
devices. All drivers reside in the os repository under the 'os/src/drivers'
subdirectory.
The general-purpose I/O (GPIO) driver located at 'gpio/imx53' implements the
revised GPIO-session interface.
The i.MX53 input driver provides support for the input devices featured on the
i.MX53 SABRE tablet. The tablet uses an Egalaxy touchscreen and Freescale's
MPR121 capacitative touch buttons. Both are supported by the new driver. The
driver is located at 'input/imx53'.
The new framebuffer driver for the i.MX53 quick-start board (QSB) as well as
the SABRE tablet comes with special support for using the
hardware overlay feature provided by the i.MX53 image processing unit (IPU)
Access to the overlay is implemented via an IPU-specific extension
of the framebuffer-session interface. To combine the driver well with
nitpicker using alpha-channels, optional support for double-buffering
is provided. The driver is located at 'framebuffer/imx53'.
As an abstraction of platform features that need to be accessed by
multiple drivers, a so-called platform driver has been introduced.
The platform driver safeguards the access to global resources such
as clocks and system-configuration bits. It can be found at 'platform/imx53'.
OMAP4 SoC
=========
The OMAP4 framebuffer driver used to support HDMI only, which was used
for connecting a display to the Pandaboard. To make the driver usable on
phones and tablets, the driver has been enhanced to support LCD output. Thanks
to Alexander Tarasikov for the patch and the insightful story about
[http://allsoftwaresucks.blogspot.com/2013/05/porting-genode-to-commercial-hardware.html - porting Genode to the B&N Nook HD+ tablet]!
USB
===
The USB driver of the 'dde_linux' repository has received substantial
improvements both feature-wise and under the hood.
First and foremost, the Linux device-driver environment, on which the
driver is based on, has been updated from kernel version 3.2 to version
3.9 as the latter version includes drivers for recent host controllers
such as DWC3 out of the box.
DWC3 is the host controller employed on the Exynos-5-based
Arndale platform for USB 3. We added the support needed to operate this
controller in XHCI mode and added support for Gigabit networking through
the ASIX AX88179 Gigabit-Ethernet Adapter as well as USB storage support.
Apart from extending the device-driver coverage, we revised the driver
internally. The back-end allocators for DMA buffers and normal memory have been
rewritten to allocate RAM more sparingly. Furthermore, we enabled the USB
driver for 64-bit x86 machines and improved the support for HID keyboards,
including the application of quirks to cherry keyboards.
*Note the change of the USB configuration*
With the addition of XHCI, the USB driver supports a growing number
of host controllers. In some situations, it is desirable to constrain the
driver to a subset of controllers only. For example, on the Arndale platform,
we desire to use a dedicated USB stack for XHCI, which operates completely
independent from the USB stack accessing USB-2. This way, gigabit networking
over USB-3 won't interfere with the operation of USB-2. To make this
possible, we added new configuration options to the USB driver.
With the new scheme, host controllers must be explicitly enabled in the
configuration. Supported config attributes are: 'uhci', 'ehci', and 'xhci'.
For example, a configuration snippet to enable UHCI and EHCI looks as
follows:
! <config uhci="yes" ehci="yes">
Updated iPXE device-driver environment
======================================
The iPXE device-driver environment was update to the most recent
iPXE upstream Git version in order to benefit from upstream improvements
of the Intel E1000 NIC driver.
Runtime environments
####################
Vancouver VMM on NOVA
=====================
Vancouver is the user-level virtual-machine monitor that accompanies the
NOVA hypervisor for hosting unmodified guest operating systems.
The most active line of development is led by Julian Stecklina at TU Dresden
via a fork called Seoul. In contrast to the original version of Vancouver,
this fork is open for outside contributions. Hence, it represents an ideal
platform for those parties with a stake in Vancouver to collaborate, i.e.,
the NUL userland, the NOVA runtime environment of TUD, and Genode.
In the current state of the transition, the Hip structure from Genode
is reused. String functions, which were formerly taken from NUL are now
provided by a stripped-down version of the C library called
'seoul_libc_support'. The nul/config.h is replaced by just using a constant
value in the one place where the file was needed.
The Genode-specific back ends of Vancouver, as largely introduced with the
previous Genode release, have been improved in several respects:
* CPUID 0x40000000: This instruction is issued by Linux when the KVM
guest support is compiled in. We have to return deterministic values to let
the Linux kernel survive.
* Replaced busy thread startup synchronization by proper locking.
* New locking scheme: We replaced the error-prone manual locking with the
use of the freshly introduced 'Synced_interface' for the motherboard and the
VCPU dispatcher. Also, all globally visible locks have been removed. They are
explicitly passed to subsystems only when needed.
* Improved PS/2 mouse back-end:
The previous version of the PS/2 mouse back end managed mouse-motion
events in a strange way, effectively throwing away most information
about the motion vector. Furthermore, the tracking of the mouse-button
states were missing. So drag-and-drop in a guest OS won't work. The new
version fixes those issues. For the transformation of input events to
PS/2 packets, the 'Genode::Register' facility is used, which greatly
simplifies the code.
L4Linux on Fiasco.OC
====================
We improved the memory management of L4Linux on Genode in two ways.
The first improvement is concerned about the upper limit of memory per Linux
instance. The corresponding discussion can be found at
[https://github.com/genodelabs/genode/issues/414 - issue #414].
We changed our L4Re emulation library to match the semantics of the original
L4Re more closely. Furthermore, we removed a heuristic in the L4Linux kernel,
which assumed that all kernel-local addresses above 0x8000000 refer to device
resources. In our version of L4Linux, there exist no MMIO resources. In
contrary, the virtual addresses above this addresses are used for normal
memory. By removing this artificial restriction with regard to the virtual
memory layout of the L4Linux kernel, we can host a larger kernel memory area.
The second improvement is concerned with the allocation of L4Linux
memory at Genode's core. Until now, L4Linux used to allocate its memory
as one contiguous RAM dataspace at core's RAM service. Core tries to
naturally align the allocation to improve the likelihood for large-page
mappings. So a dataspace is likely to be physically located at a
power-of-two boundary larger or equal than the dataspace size. For example,
the allocation of a 100 MiB RAM dataspace for a Linux instance will
be located at a 128 MiB boundary. If multiple of such allocations happen
sub-sequentially, this allocation strategy results in 28 MiB gaps between
100 MiB dataspaces. This memory cannot be used for large contiguous
allocations anymore. So even if the available memory capacity is far
larger than 100 MiB, an allocation of a 100 MiB block may fail.
To relieve this problem, we weakened the requirement for contiguous memory
by assembling L4Linux memory from multiple chunks of small dataspaces.
For example, by using a chunk size of 16 MiB, core's best-fit allocator
will have a better chance to find a more suited position for allocation
when aligning the block to a 16 MiB boundary compared to the allocation
of a larger block. Furthermore, slack memory can be used more efficiently
because smaller gaps (such as a 20 MiB gap) remain to be usable for L4Linux.
The discussion of this topic and the individual patch can be found at
[https://github.com/genodelabs/genode/issues/695 - issue #695].
Furthermore, the L4Linux block driver has been improved to support large
partitions.
Platforms
#########
Execution on bare hardware (base-hw)
====================================
Raspberry Pi
~~~~~~~~~~~~
Principal support for the Raspberry Pi platform has been added to the base-hw
kernel. The popular Raspberry Pi board is based on an ARMv6 Broadcom BCM2835
SoC. The current scope of the platform support comprises:
* IRQ controller driver: Because the interrupt controller uses a cascade of
registers, we settled on the following IRQ enumeration scheme.
IRQ numbers 0..7 refer to the basic IRQs.
IRQ numbers 8..39 refer to GPU IRQs 0..31.
IRQ numbers 40..71 refer to GPU IRQs 32..63.
* The kernel employs the so-called system timer for the preemptive scheduling.
* Core's LOG messages are printed over the PL011-based UART.
* The user-level timer driver uses the so-called ARM timer, which is a
slightly modified SP804 timer device.
Up to this point, a few device driver are missing to use Genode on the
Raspberry Pi in practice, most notably USB.
To build and run Genode on the Raspberry Pi, create a new build directory
via the 'create_builddir' tool, specifying 'hw_rpi' as platform.
User-level timer driver for Arndale platform
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
By adding our new Exynos 5250 PWM timer driver, the base-hw kernel can now
be used for executing meaningful scenarios on the Arndale board including
the USB stack and networking.
Linux
=====
Until now, Genode on Linux supported x86-based platforms only.
The newly added 'linux_arm' platform clears the way to run Genode directly on
Linux-based ARM platforms. Genode's entire software stack is supported,
including the dynamic linker, graphical applications, and Qt4.
As a known limitations, the libc 'setjmp()'/'longjmp()' doesn't currently
save/restore floating point registers.
Build system and tools
######################
The run tool has been enhanced as detailed in Section
[Automated quality-assurance testing].

995
doc/release_notes-13-08.txt
View File

@@ -1,995 +0,0 @@
===============================================
Release notes for the Genode OS Framework 13.08
===============================================
Genode Labs
The release of version 13.08 marks the 5th anniversary of the Genode OS
framework. We celebrate this anniversary with the addition of three major
features that we have much longed for, namely the port of Qt5 to Genode,
profound multi-processor support, and a light-weight event tracing
framework. Additionally, the new version comes with new device drivers for
SATA 3.0 and power management for the Exynos-5 SoC, improved virtualization
support on NOVA on x86, updated kernels, and integrity checks for
downloaded 3rd-party source code.
Over the course of the past five years, Genode's development was primarily
motivated by adding and cultivating features to make the framework fit for as
many application areas as possible. Now that we have a critical mass of
features, the focus on mere functionality does not suffice anymore. The
question of what Genode can do ultimately turns into the question of how well
Genode can do something: How stable is a certain workload? How does networking
perform? How does it scale to multi-processor systems? Because we are lacking
concise answers to these kind of questions, we have to investigate.
When talking about stability, our recently introduced automated testing
infrastructure makes us more confident than ever. Each night, over 200
automated tests are performed, covering various kernels and several hardware
platforms. All those tests are publicly available in the form of so-called run
scripts and are under continues development.
Regarding performance investigations, recently we have begun to benchmark
application performance focusing on network throughput. Interestingly, our
measurements reveal significant differences between the used kernels, but
also shortcomings in our software stack. For example, currently we see that
our version of lwIP performs poorly with gigabit networking. To thoroughly
investigate such performance issues, the current version adds support
for tracing the behaviour of Genode components. This will allow us to get a
profound understanding of all inter-component interaction that are on the
critical path for the performance of complex application-level workloads.
Thanks to the Genode architecture, we could come up with a strikingly simple,
yet powerful design for a tracing facility. Section
[Light-weight event tracing] explains how it works.
When it comes to multi-processor scalability, we used to shy away from such
inquiries because, honestly, we haven't paid much consideration to it. This
view has changed by now. With the current release, we implemented the
management of CPU affinities right into the heart of the framework, i.e.,
Genode's session concept. Additionally, we cracked a damn hard nut by enabling
Genode to use multiple CPUs on the NOVA hypervisor. This kernel is by far the
most advanced Open-Source microkernel for the x86 architecture. However,
NOVA's MP model seemed to inherently contradict with the API design of Genode.
Fortunately, we found a fairly elegant way to go forward and we're able to
tame the beast. Section [Enhanced multi-processor support] goes into more
detail.
Functionality-wise, we always considered the availability of Qt on Genode as a
big asset. With the current release, we are happy to announce that we finally
made the switch from Qt4 to Qt5. Section [Qt5 available on all kernels] gives
insights into the challenges that we faced during porting work.
In addition to those highlights, the new version comes with improvements all
over the place. To name a few, there are improved support for POSIX threads,
updated device drivers, an updated version of the Fiasco.OC kernel and
L4Linux, and new device drivers for Exynos-5. Finally, the problem of
verifying the integrity of downloaded 3rd-party source codes has been
addressed.
Qt5 available on all kernels
############################
Since its integration with Genode version 9.02, Qt4 is regarded as
one of the most prominent features of the framework. For users, combining Qt
with Genode makes the world of sophisticated GUI-based end-user applications
available on various microkernels. For Genode developers, Qt represents by far
the most complex work load natively executed on top of the framework API,
thereby stressing the underlying system in any way imaginable. We have been
keeping an eye on Qt version 5 for a while and highly anticipate the direction
where Qt is heading. We think that the time is right to leave Qt4 behind to
embrace Qt5 for Genode.
For the time being, both Qt4 and Qt5 are available for Genode, but Qt4 is
declared as deprecated and will be removed with the upcoming version 13.11.
Since Qt5 is almost API compatible to Qt4, the migration path is relatively
smooth. So we recommend to move your applications over to Qt5 during the
next release cycle.
In the following, we briefly describe the challenges we faced while adding Qt5
support to Genode, point you to the place where to find Qt5 in the source
tree, and give quick-start instructions for getting a Qt5 application
scenario running.
We found that the biggest architectural difference between version 4 and
version 5 is the use of the so-called Qt Platform Abstraction (QPA) interface,
which replaces the former Qt Window System (QWS).
Moving from QWS to QPA
======================
With Qt4, we relied on QWS
to perform the window handling. A Qt4 application used to create a session to
Genode's GUI server (called nitpicker) and applied its graphical output onto a
virtual framebuffer. The virtual framebuffer was not visible per se. To make
portions of the virtual framebuffer visible on screen, the application had to
create so-called nitpicker views. A view is a rectangular area of the physical
screen that displays a (portion of) the virtual framebuffer. The position,
size, and stacking order of views is managed by the application. For each Qt
window, the application would simply create a corresponding nitpicker view and
maintain the consistency of the view with the geometry of the Qt window. Even
though each Qt application seemingly operated as a full-screen application
with the windows managed by the application-local QWS, the use of nitpicker
views still allowed the integration of any number of Qt applications into one
windowed environment.
With the advent of compositing window managers, the typical way of how an
application interacts with the window system of the OS changed. Whereas
old-generation GUIs relied on a tight interplay of the application with the
window server in order to re-generate newly exposed window regions whenever
needed (e.g., revealing the window content previously covered by another
window), the modern model of a GUI server keeps all pixels of all windows in
memory regardless of whether the window is visible or covered by other
windows. The use of one pixel buffer per window seems wasteful with respect to
memory usage when many large windows are overlapping each other. On the other
hand, this technique largely simplifies GUI servers and makes the
implementation of fancy effects, like translucent windows, straight forward.
Since memory is cheap, the Qt developers abandoned the old method and fully
embraced the buffer-per-window approach by the means of QPA.
For Genode, we faced the challenge that we don't have a window server
in the usual sense. With nitpicker, we have a GUI server, but with a more
radical design. In particular, nitpicker leaves the management of window
geometries and stacking to the client. In contrast, QPA expects the window
system to provide both means for a user to interactively change the window
layout and a way for an application to define the properties (such as the
geometry, title, and visibility) of its windows.
The obviously missing piece was the software component that deals with window
controls. Fortunately, we already have a bunch of native nitpicker applications
that come with client-side window controls, in particular the so-called liquid
framebuffer (liquid_fb). This nitpicker client presents a virtual framebuffer
in form of a proper window on screen and, in turn, provides a framebuffer and
input service. These services can be used by other Genode processes, for
example, another nested instance of nitpicker.
This way, liquid_fb lends itself to be the interface between the nitpicker
GUI server and QPA.
For each QPA window, the application creates a new liquid_fb instance as a
child process. The liquid_fb instance will request a dedicated nitpicker
session, which gets routed through the application towards the parent of the
application, which eventually routes the request to the nitpicker GUI server.
Finally, the liquid_fb instance announces its input and framebuffer services
to its parent, which happens to be the application. Now, the application is
able to use those services in order to access the window. Because the
liquid_fb instances are children of the application, the application can
impose control over those. In particular, it can update the liquid_fb
configuration including the window geometry and title at any time. Thanks to
Genode's dynamic reconfiguration mechanism, the liquid_fb instances are able
to promptly respond to such reconfigurations.
Combined, those mechanisms give the application a way to receive user input
(via the input services provided by the liquid_fb instances), perform
graphical output (via the virtual framebuffers provided by the liquid_fb
instances), and define window properties (by dynamically changing the
respective liquid_fb configurations). At the same time, the user can use
liquid_fb's window controls to move, stack, and resize application windows as
expected.
[image qt5_screenshot]
Steps of porting Qt5
====================
Besides the switch to QPA, the second major change was related to the build
system. For the porting work, we use a Linux host system to obtain the
starting point for the build rules. The Qt4 build system would initially
generate all Makefiles, which could be inspected and processed at once. In
contrast, Qt5 generates Makefiles during the build process whenever needed.
When having configured Qt for Genode, however, the build on Linux will
ultimately fail. So the much-desired intermediate Makefiles won't be created.
The solution was to have 'configure' invoke 'qmake -r' instead of 'qmake'.
This way, qmake project files will be processed recursively. A few additional
tweaks were needed to avoid qmake from backing out because of missing
dependencies (qt5_configuration.patch). To enable the build of the Qt tools
out of tree, qmake-specific files had to be slightly adapted
(qt5_tools.patch). Furthermore, qtwebkit turned out to use code-generation
tools quite extensively during the build process. On Genode, we perform this
step during the 'make prepare' phase when downloading and integrating the Qt
source code with the Genode source tree.
For building Qt5 on Genode, we hit two problems. First, qtwebkit depends on
the ICU (International Components for Unicode) library, which was promptly
ported and can be found in the libports repository. Second, qtwebkit
apparently dropped the support of the 'QThread' API in favor of
POSIX-thread support only. For this reason, we had to extend the coverage
of Genode's pthread library to fulfill the needs of qtwebkit.
Once built, we entered the territory of debugging problems at runtime.
* We hit a memory-corruption problem caused by an assumption of 'QArrayData'
with regard to the alignment of memory allocated via malloc. As a
work-around, we weakened the assumptions to 4-byte alignment
(qt5_qarraydata.patch).
* Page faults in QWidgetAnimator caused by
use-after-free problems. Those could be alleviated by adding pointer
checks (qt5_qwidgetanimator.patch).
* Page faults caused by the slot function 'QWidgetWindow::updateObjectName()'
with a 'this' pointer of an incompatible type 'QDesktopWidget*'.
As a workaround, we avoid this condition by delegating the
'QWidgetWindow::event()' that happened to trigger the slot method to
'QWindow' (base class of 'QWidgetWindow') rather than to a
'QDesktopWidget' member (qt5_qwidgetwindow.patch).
* We observed that Arora presented web sites incomplete, or including HTTP
headers. During the evaluation of HTTP data, a signal was sent to another
thread, which activated a "user provided download buffer" for optimization
purposes. On Linux, the receiving thread was immediately scheduled and
everything went fine. However, on some kernels used by Genode, scheduling
is different, so that the original thread continued to execute a bit longer,
ultimately triggering a race condition. As a workaround, we disabled the
"user provided download buffer" optimization.
* Page faults in the JavaScript engine of Webkit. The JavaScript
'RegExp.exec()' function returned invalid string objects. We worked around
this issue by deactivating the JIT compiler for the processing of
regular expressions (ENABLE_YARR_JIT).
The current state of the Qt5 port is fairly complete. It covers the core, gui,
jscore, network, script, scriptclassic, sql, ui, webcore, webkit, widgets,
wtf, and xml modules. That said, there are a few known limitations and
differences compared to Qt4. First, the use of one liquid_fb instance per
window consumes more memory compared to the use of QWS in Qt4. Furthermore,
external window movements are not recognized by our QPA implementation yet.
This can cause popup menus to appear at unexpected positions. Key repeat is
not yet handled. The 'QNitpickerViewWidget' is not yet adapted to Qt5. For this
reason, qt_avplay is not working yet.
Test drive
==========
Unlike Qt4, which was hosted in the dedicated 'qt4' repository, Qt5 is
integrated in the libports repository. It can be downloaded and integrated
into the Genode build system by issuing 'make prepare' from within the
libports repository. The Qt5 versions of the known Qt examples are located at
libports/src/app/qt5. Ready-to-use run scripts for those examples are available
at libports/run.
Migration away from Qt4 to Qt5
==============================
The support for Qt4 for Genode has been declared as deprecated. By default,
it's use is inhibited to avoid name aliasing problems between both versions.
Any attempt to build a qt4-based target will result in a message:
!Skip target app/qt_launchpad because it requires qt4_deprecated
To re-enable the use of Qt4, the SPEC value qt4_deprecated must be defined
manually for the build directory:
!echo "SPECS += qt4_deprecated" >> etc/specs.conf
We will keep the qt4 repository in the source tree during the current
release cycle. It will be removed with version 13.11.
Light-weight event tracing
##########################
With Genode application scenarios getting increasingly sophisticated,
the need for thorough performance analysis has come into spotlight.
Such scenarios entail the interaction of many components.
For example, with our recent work on optimizing network performance, we
have to consider several possible attack points:
* Device driver: Is the device operating in the proper mode? Are there
CPU-intensive operations such as allocations within the critical path?
* Interface to the device driver: How frequent are context switches between
client and device driver? Is the interface designed appropriately for
the access patterns?
* TCP/IP stack: How does the data flow from the raw packet level to the
socket level? How dominant are synchronization costs between the involved
threads? Are there costly in-band operations performed, e.g., dynamic
memory allocations per packet?
* C runtime: How does integration of the TCP/IP stack with the C runtime
work, for example how does the socket API interfere with timeout
handling during 'select' calls?
* Networking application
* Timer server: How often is the timer consulted by the involved components?
What is the granularity of timeouts and thereby the associated costs for
handling them?
* Interaction with core: What is the profile of the component's interaction
with core's low-level services?
This example is just an illustration. Most real-world performance-critical
scenarios have a similar or even larger scope. With our traditional
tools, it is hardly possible to gather a holistic view of the scenario. Hence,
finding performance bottlenecks tends to be a series of hit-and-miss
experiments, which is a tiresome and costly endeavour.
To overcome this situation, we need the ability to gather traces of component
interactions. Therefore, we started investigating the design of a tracing
facility for Genode one year ago while posing the following requirements:
* Negligible impact on the performance, no side effects:
For example, performing a system call per traced event
is out of question because this would severely influence the flow of
control (as the system call may trigger the kernel to take a scheduling
decision) and the execution time of the traced code, not to speak of
the TLB and cache footprint.
* Kernel independence: We want to use the same tracing facility across
all supported base platforms.
* Accountability of resources: It must be clearly defined where the
resources for trace buffers come from. Ideally, the tracing tool should be
able to dimension the buffers according to its needs and, in turn, pay for
the buffers.
* Suitable level of abstraction: Only if the trace contains information at
the right level of abstraction, it can be interpreted for large scenarios.
A counter example is the in-kernel trace buffer of the Fiasco.OC kernel,
which logs kernel-internal object names and a few message words when tracing
IPC messages, but makes it almost impossible to map this low-level
information to the abstraction of the control flow of RPC calls. In
contrast, we'd like to capture the names of invoked RPC calls (which is an
abstraction level the kernel is not aware of). This requirement implies the
need to have useful trace points generated automatically. Ideally those
trace points should cover all interactions of a component with the outside
world.
* (Re-)definition of tracing policies at runtime: The
question of which information to gather when a trace point is passed
should not be solely defined at compile time. Instead of changing static
instrumentations in the code, we'd prefer to have a way to configure
the level of detail and possible conditions for capturing events at runtime,
similar to dtrace. This way, a series of different hypotheses could be
tested by just changing the tracing policy instead of re-building and
rebooting the entire scenario.
* Straight-forward implementation: We found that most existing tracing
solutions are complicated. For example, dtrace comes with a virtual
machine for the sandboxed interpretation of policy code. Another typical
source of complexity is the synchronization of trace-buffer accesses.
Because for Genode, low TCB complexity is of utmost importance, the
simplicity of the implementation is the prerequisite to make it an
integral part of the base system.
* Support for both online and offline analysis of traces.
We are happy to report to have come up with a design that meets all those
requirements thanks to the architecture of Genode. In the following, we
present the key aspects of the design.
The tracing facility comes in the form of a new TRACE service implemented
in core. Using this service, a TRACE client can gather information about
available tracing subjects (existing or no-longer existing threads),
define trace buffers and policies and assign those to tracing subjects,
obtain access to trace-buffer contents, and control the tracing state
of tracing subjects. When a new thread is created via a CPU session, the
thread gets registered at a global registry of potential tracing sources. Each
TRACE service manages a session-local registry of so-called trace subjects.
When requested by the TRACE client, it queries new tracing sources from the
source registry and obtains references to the corresponding threads. This way,
the TRACE session becomes able to control the thread's tracing state.
To keep the tracing overhead as low as possible, we assign a separate trace
buffer to each individually traced thread. The trace buffer is a shared memory
block mapped to the virtual address space of the thread's process. Capturing
an event comes down to a write operation into the thread-local buffer. Because
of the use of shared memory for the trace buffer, no system call is needed and
because the buffer is local to the traced thread, there is no need for
synchronizing the access to the buffer. When no tracing is active, a thread
has no trace buffer. The buffer gets installed only when tracing is started.
The buffer is not installed magically from the outside of the traced process
but from the traced thread itself when passing a trace point. To detect
whether to install a new trace buffer, there exists a so-called trace-control
dataspace shared between the traced process and its CPU service. This
dataspace contains control bits for each thread created via the CPU session.
The control bits are evaluated each time a trace point is passed by the
thread. When the thread detects a change of the tracing state, it actively
requests the new trace buffer from the CPU session and installs it into its
address space. The same technique is used for loading the code for tracing
policies into the traced process. The traced thread actively checks for
policy-version updates by evaluating the trace-control bits. If an update is
detected, the new policy code is requested from the CPU session. The policy
code comes in the form of position-independent code, which gets mapped into
the traced thread's address space by the traced thread itself. Once mapped,
a trace point will call the policy code. When called, the policy
module code returns the data to be captured into the trace buffer. The
relationship between the trace monitor (the client of TRACE service), core's
TRACE service, core's CPU service, and the traced process is depicted in
Figure [trace_control].
[image trace_control]
There is one trace-control dataspace per CPU session, which gets accounted
to the CPU session's quota. The resources needed for the
trace-buffer dataspaces and the policy dataspaces are paid-for by the
TRACE client. On session creation, the TRACE client can specify the amount
of its own RAM quota to be donated to the TRACE service in core. This
enables the TRACE client to define trace buffers and policies of arbitrary
sizes, limited only by its own RAM quota.
In Genode, the interaction of a process with its outside world is
characterized by its use of inter-process communication, namely synchronous
RPC, signals, and access to shared memory. For the former two types of
inter-process communication, Genode generates trace points automatically. RPC
clients generate trace points when an RPC call is issued and when a call
returned. RPC servers generate trace points in the RPC dispatcher, capturing
incoming RPC requests as well as RPC replies. Thanks to Genode's RPC
framework, we are able to capture the names of the RPC functions in the RPC
trace points. This information is obtained from the declarations of the RPC
interfaces. For signals, trace points are generated for submitting and
receiving signals. Those trace points form a useful base line for gathering
tracing data. In addition, manual trace points can be inserted into the code.
State of the implementation
===========================
The implementation of Genode's tracing facility is surprisingly low complex.
The addition to the base system (core as well as the base library) are
merely 1500 lines of code. The mechanism works across all base platforms.
Because the TRACE client provides the policy code and trace buffer to the
traced thread, the TRACE client imposes ultimate control over the traced
thread. In contrast to dtrace, which sandboxes the trace policy, we express
the policy module in the form of code executed in the context of the traced
thread. However, in contrast to dtrace, such code is never loaded into a large
monolithic kernel, but solely into the individually traced processes. So the
risk of a misbehaving policy is constrained to the traced process.
In the current form, the TRACE service of core should be considered as a
privileged service because the trace-subject namespace of each session
contains all threads of the system. Therefore, TRACE sessions should be routed
only for trusted processes. In the future, we plan to constrain the
namespaces for tracing subjects per TRACE session.
The TRACE session interface is located at base/include/trace_session/.
A simple example for using the service is available at os/src/test/trace/
and is accompanied with the run script os/run/trace.run. The test
demonstrates the TRACE session interface by gathering a trace of a thread
running locally in its address space.
Enhanced multi-processor support
################################
Multi-processor (MP) support is one of those features that most users take for
granted. MP systems are so ubiquitous, even on mobile platforms, that a
limitation to utilizing a single CPU only is almost a fallacy.
That said, MP support in operating systems is hard to get right. For this
reason, we successively deferred the topic on the agenda of Genode's
road map.
For some base platforms such as the Linux or Codezero kernels, Genode
always used to support SMP because the kernel would manage the affinity
of threads to CPU cores transparently to the user-level process. So on
these kernels, there was no need to add special support into the framework.
However, on most microkernels, the situation is vastly different. The
developers of such kernels try hard to avoid complexity in the kernel and
rightfully argue that in-kernel affinity management would contribute to kernel
complexity. Another argument is that, in contrast to monolithic kernels that
have a global view on the system and an "understanding" of the concerns of the
user processes, a microkernel is pretty clueless when it comes to the roles
and behaviours of individual user-level threads. Not knowing whether a thread
works as a device driver, an interactive program, or a batch process, the
microkernel is not in the position to form a reasonably useful model of the
world, onto which it could intelligently apply scheduling and affinity
heuristics. In fact, from the perspective of a microkernel, each thread does
nothing else than sending and receiving messages, and causing page faults.
For these reasons, microkernel developers tend to provide the bootstrapping
procedure for the physical CPUs and a basic mechanism to assign
threads to CPUs but push the rest of the problem to the user space, i.e.,
Genode. The most straight-forward way would make all physical CPUs visible
to all processes and require the user or system integrator to assign
physical CPUs when a thread is created. However, on the recursively
structured Genode system, we virtualize resources at each level, which
calls for a different approach. Section [Management of CPU affinities]
explains our solution.
When it comes to inter-process communication on MP systems, there is a
certain diversity among the kernel designs. Some kernels allow the user land
to use synchronous IPC between arbitrary threads, regardless of whether both
communication partners reside on the same CPU or on two different CPUs. This
convenient model is provided by Fiasco.OC. However, other kernels do not offer
any synchronous IPC mechanism across CPU cores at all, NOVA being a poster
child of this school of thought. If a user land is specifically designed for
a particular kernel, those peculiarities can be just delegated to the
application developers. For example, the NOVA user land called NUL is designed
such that a recipient of IPC messages spawns a dedicated thread on each
physical CPU. In contrast, Genode is meant to provide a unified API that
works well across various different kernels. To go forward, we had four
options:
# Not fully supporting the entity of API semantics across all base platforms.
For example, we could stick with the RPC API for synchronous communication
between threads. Programs just would happen to fail on some base platforms
when the called server resides on a different CPU. This would effectively
push the problem to the system integrator. The downside would be the
sacrifice of Genode's nice feature that a program developed
on one kernel usually works well on other kernels without any changes.
# Impose the semantics provided by the most restrictive kernel onto all
users of the Genode API. Whereas this approach would facilitate that
programs behave consistently across all base platforms, the restrictions
would be artificially imposed onto all Genode users, in particular the
users of kernels with less restrictions. Of course, we don't change
the Genode API lightheartedly, which attributes to our hesitance to go
into this direction.
# Hiding kernel restrictions behind the Genode API. This approach could come
in many different shapes. For example, Genode could transparently spawn a
thread on each CPU when a single RPC entrypoint gets created, following
the model of NUL. Or Genode could emulate synchronous IPC using the core
process as a proxy.
# Adapting the kernel to the requirements of Genode. That is, persuading
kernel developers to implement the features we find convenient, i.e., adding
a cross-CPU IPC feature to NOVA. History shows that our track record in
doing that is not stellar.
Because each of those options is like opening a different can of worms, we
used to defer the solution of the problem. Fortunately, however, we finally
kicked off a series of practical experiments, which led to a fairly elegant
solution, which is detailed in Section
[Adding multi-processor support to Genode on NOVA].
Management of CPU affinities
============================
In line with our experience of supporting
[http://www.genode.org/documentation/release-notes/10.02#Real-time_priorities - real-time priorities]
in version 10.02, we were seeking a way to express CPU affinities such that
Genode's recursive nature gets preserved and facilitated. Dealing with
physical CPU numbers would contradict with this mission. Our solution
is based on the observation that most MP systems have topologies that can
be represented on a two-dimensional coordinate system. CPU nodes close
to each other are expected to have closer relationship than distant nodes.
In a large-scale MP system, it is natural to assign clusters of closely
related nodes to a given workload. Genode's architecture is based on the
idea to recursively virtualize resources and thereby lends itself to the
idea to apply this successive virtualization to the problem of clustering
CPU nodes.
In our solution, each process has a process-local view on a so-called affinity
space, which is a two-dimensional coordinate space. If the process creates a
new subsystem, it can assign a portion of its own affinity space to the new
subsystem by imposing a rectangular affinity location to the subsystem's CPU
session. Figure [affinity_space] illustrates the idea.
[image affinity_space]
Following from the expression of affinities as a rectangular location within a
process-local affinity space, the assignment of subsystems to CPU nodes
consists of two parts, the definition of the affinity space dimensions as used
for the process and the association of sub systems with affinity locations
(relative to the affinity space). For the init process, the affinity space is
configured as a sub node of the config node. For example, the following
declaration describes an affinity space of 4x2:
! <config>
! ...
! <affinity-space width="4" height="2" />
! ...
! </config>
Subsystems can be constrained to parts of the affinity space using the
'<affinity>' sub node of a '<start>' entry:
! <config>
! ...
! <start name="loader">
! <affinity xpos="0" ypos="1" width="2" height="1" />
! ...
! </start>
! ...
! </config>
As illustrated by this example, the numbers used in the declarations for this
instance of the init process are not directly related to physical CPUs. If
the machine has just two cores, init's affinity space would be mapped
to the range [0,1] of physical CPUs. However, in a machine
with 16x16 CPUs, the loader would obtain 8x8 CPUs with the upper-left
CPU at position (4,0). Once a CPU session got created, the CPU client can
request the physical affinity space that was assigned to the CPU session
via the 'Cpu_session::affinity()' function. Threads of this CPU session
can be assigned to those physical CPUs via the 'Cpu_session::affinity()'
function, specifying a location relative to the CPU-session's affinity space.
Adding multi-processor support to Genode on NOVA
================================================
The NOVA kernel has been supporting MP systems for a long time. However
Genode did not leverage this capability until now. The main reason was that
the kernel does not provide - intentionally by the kernel developer - the
possibility to perform synchronous IPC between threads residing on different
CPUs.
To cope with this situation, Genode servers and clients would need to make
sure to have at least one thread on a common CPU in order to communicate.
Additionally, shared memory and semaphores could be used to communicate across
CPU cores. Both options would require rather fundamental changes to the Genode
base framework and the API. An exploration of this direction should in any
case be pursued in evolutionary steps rather than as one big change, also
taking into account that other kernels do not impose such hard requirements on
inter-CPU communication. To tackle the challenge, we conducted a series of
experiments to add some kind of cross-CPU IPC support to Genode/NOVA.
As a general implication of the missing inter-CPU IPC, messages between
communication partners that use disjoint CPUs must take an indirection through
a proxy process that has threads running on both CPUs involved. The sender
would send the message to a proxy thread on its local CPU, the proxy process
would transfer the message locally to the CPU of the receiver by using
process-local communication, and the proxy thread on the receiving CPU would
deliver the message to the actual destination. We came up with three options
to implement this idea prototypically:
# Core plays the role of the proxy because it naturally has access to all
CPUs and emulates cross-CPU IPC using the thread abstractions of the
Genode API.
# Core plays the role of the proxy but uses NOVA system calls directly
rather than Genode's thread abstraction.
# The NOVA kernel acts as the proxy and emulates cross-CPU IPC directly
in the kernel.
After having implemented the first prototypes, we reached the following
conclusions.
For options 1 and 2 where core provides this service: If a client can not
issue a local CPU IPC, it asks core - actually the pager of the client
thread - to perform the IPC request. Core then spawns or reuses a proxy thread
on the target CPU and performs the actual IPC on behalf of the client. Option
1 and 2 only differ in respect to code size and the question to whom to
account the required resources - since a proxy thread needs a stack and some
capability selectors.
As one big issue for option 1 and 2, we found that in order to delegate
capabilities during the cross-CPU IPC, core has to receive capability mappings
to delegate them to the target thread. However, core has no means to know
whether the capabilities must be maintained in core or not. If a capability is
already present in the target process, the kernel would just translate the
capability to the target's capability name space. So core wouldn't need to
keep it. In the other case where the target receives a prior unknown
capability, the kernel creates a new mapping. Because the mapping gets
established by the proxy in core, core must not free the capability.
Otherwise, the mapping would disappear in the target process. This means that
the use of core as a proxy ultimately leads to leaking kernel resources
because core needs to keep all transferred capabilities, just for the case a
new mapping got established.
For option 3, the same general functionality as for option 1 and 2 is
implemented in the kernel instead of core. If a local CPU IPC call
fails because of a BAD_CPU kernel error code, the cross-CPU IPC extension
will be used. The kernel extension creates - similar as to option 1 and 2 - a
semaphore (SM), a thread (EC), and a scheduling context (SC) on the remote
CPU and lets it run on behalf of the caller thread. The caller thread
gets suspended by blocking on the created semaphore until the remote EC has
finished the IPC. The remote proxy EC reuses the UTCB of the suspended caller
thread as is and issues the IPC call. When the proxy EC returns, it wakes up
the caller via the semaphore. Finally, the proxy EC and SC de-schedule
themselves and the resources get to be destroyed later on by the kernel's RCU
mechanism. Finally, when the caller thread got woken up, it takes care to
initiate the deconstruction of the semaphore.
The main advantage of option 3 compared to options 1 and 2 is that we don't
have to keep and track the capability delegations during a cross-CPU IPC.
Furthermore, we do not have potentially up to two additional address space
switches per cross-CPU IPC (from client to core and core to the server).
Additionally, the UTCB of the caller is reused by the proxy EC and does not
need to be allocated separately as for option 1 and 2.
For these reasons, we decided to go for the third option. From Genode's API
point of view, the use of cross-CPU IPC is completely transparent. Combined
with the affinity management described in the previous section, Genode/NOVA
just works on MP systems.
As a simple example for using Genode on MP systems, there is a ready-to-use
run script available at base/run/affinity.run.
Base framework
##############
Affinity propagation in parent, root, and RPC entrypoint interfaces
===================================================================
To support the propagation of CPU affinities with session requests, the
parent and root interfaces had to be changed. The 'Parent::Session'
and 'Root::Session' take the affinity of the session as a new argument.
The affinity argument contains both the dimensions of the affinity space
used by the session and the session's designated affinity location within
the space. The corresponding type definitions can be found at
base/affinity.h.
Normally, the 'Parent::Session' function is not used directly but indirectly
through the construction of a so-called connection object, which represents an
open session. For each session type there is a corresponding connection type,
which takes care of assembling the session-argument string by using the
'Connection::session()' convenience function. To maintain API compatibility,
we kept the signature of the existing 'Connection::session()' function using a
default affinity and added a new overload that takes the affinity as
additional argument. Currently, this overload is used in
cpu_session/connection.h.
For expressing the affinities of RPC entrypoints to CPUs within the affinity
space of the server process, the 'Rpc_entrypoint' takes the desired affinity
location of the entrypoint as additional argument. For upholding API
compatibility, the affinity argument is optional.
CPU session interface
=====================
The CPU session interface underwent changes to accommodate the new event
tracing infrastructure and the CPU affinity management.
Originally the 'Cpu_session::num_cpus()' function could be used to
determine the number of CPUs available to the session. This function
has been replaced by the new 'affinity_space' function, which returns the
bounds of the CPU session's physical affinity space. In the simplest case of
an SMP machine, the affinity space is one-dimensional where the width
corresponds to the number of CPUs. The 'affinity' function, which is used to
bind a thread to a specified CPU, has been changed to take an affinity
location as argument. This way, the caller can principally express the
affiliation of the thread with multiple CPUs to guide load-balancing in a CPU
service.
New TRACE session interface
===========================
The new event tracing mechanism as described in Section
[Light-weight event tracing] is exposed to Genode processes in the form
of the TRACE service provided by core. The new session interface
is located under base/include/trace_session/. In addition to the new session
interface, the CPU session interface has been extended with functions for
obtaining the trace-control dataspace for the session as well as the trace
buffer and trace policy for a given thread.
Low-level OS infrastructure
###########################
Event-driven operation of NIC bridge
====================================
The NIC bridge component multiplexes one physical network device among
multiple clients. It enables us to multiplex networking on the network-packet
level rather than the socket level and thereby take TCP/IP out of the
critical software stack for isolating network applications. As it
represents an indirection in the flow of all networking packets, its
performance is important.
The original version of NIC bridge was heavily multi-threaded. In addition to
the main thread, a timer thread, and a thread for interacting with the NIC
driver, it employed one dedicated thread per client. By merging those flows of
control into a single thread, we were able to significantly reduce the number
of context switches and improve data locality. These changes reduced the
impact of the NIC bridge on the packet throughput from 25% to 10%.
Improved POSIX thread support
=============================
To accommodate qtwebkit, we had to extend Genode's pthread library with
working implementations of condition variables, mutexes, and thread-local
storage. The implemented functions are attr_init, attr_destroy, attr_getstack,
attr_get_np, equal, mutex_attr, mutexattr_init, mutexattr_destroy,
mutexattr_settype, mutex_init, mutex_destroy, mutex_lock, mutex_unlock,
cond_init, cond_timedwait, cond_wait, cond_signal, cond_broadcast, key_create,
setspecific, and getspecific.
Device drivers
##############
SATA 3.0 on Exynos 5
====================
The previous release featured the initial version of our SATA 3.0 driver for
the Exynos 5 platform. This driver located at os/src/drivers/ahci/exynos5 has
reached a fully functional state by now. It supports UDMA-133 with up to
6 GBit/s.
For driver development, we set the goal to reach a performance equal to
the Linux kernel. To achieve that goal, we had to make sure to
operate the controller and the disks in the same ways as Linux does.
For this reason, we modeled our driver closely after the behaviour of the
Linux driver. That is, we gathered traces of I/O transactions to determine the
initialization steps and the request patterns that Linux performs to access
the device, and used those behavioral traces as a guide for our
implementation. Through step-by-step analysis of the traces, we not only
succeeded to operate the device in the proper modes, but we also found
opportunities for further optimization, in particular regarding the error
recovery implementation.
This approach turned out to be successful. We measured that our driver
generally operates as fast (and in some cases even significantly faster)
than the Linux driver on solid-state disks as well as on hard disks.
Dynamic CPU frequency scaling for Exynos 5
==========================================
As the Samsung Exynos-5 SoC is primarily targeted at mobile platforms,
power management is an inherent concern. Until now, Genode did not pay
much attention to power management though. For example, we completely
left out the topic from the scope of the OMAP4 support. With the current
release, we took the first steps towards proper power management on ARM-based
platforms in general, and the Exynos-5-based Arndale platform in particular.
First, we introduced a general interface to regulate clocks and voltages.
Priorly, each driver did its own part of configuring clock and power control
registers. The more device drivers were developed, the higher were the chances
that they interfere when accessing those clock, or power units.
The newly introduced "Regulator" interface provides the possibility to enable
or disable, and to set or get the level of a regulator. A regulator might be
a clock for a specific device (such as a CPU) or a voltage regulator.
For the Arndale board, an exemplary implementation of the regulator interface
exists in the form of the platform driver. It can be found at
os/src/drivers/platform/arndale. Currently, the driver implements
clock regulators for the CPU, the USB 2.0 and USB 3.0 host controller, the
eMMC controller, and the SATA controller. Moreover, it provides power
regulators for SATA, USB 2.0, and USB 3.0 host controllers. The selection of
regulators is dependent on the availability of drivers for the platform.
Otherwise it wouldn't be possible to test that clock and power state doesn't
affect the device.
Apart from providing regulators needed by certain device drivers, we
implemented a clock regulator for the CPU that allows changing the CPU
frequency dynamically and thereby giving the opportunity to scale down
voltage and power consumption. The possible values range from 200 MHz to
1.7 GHz whereby the last value isn't recommended and might provoke system
crashes due to overheating. When using Genode's platform driver for Arndale
it sets CPU clock speed to 1.6 GHz by default. When reducing
the clock speed to the lowest level, we observed a power consumption
reduction of approximately 3 Watt. Besides reducing dynamic power consumption
by regulating the CPU clock frequency, we also explored the gating of the clock
management and power management to further reduce power consumption.
With the CPU frequency scaling in place, we started to close all clock gates
not currently in use. When the platform driver for the Arndale board gets
initialized, it closes everything. If a device driver enables its clock
regulator, all necessary clock gates for the device's clock are opened. This
action saves about 0.7 Watt. The initial closing of all unnecessary power
gates was much more effective. Again, everything not essential for the working
of the kernel is disabled on startup. When a driver enables its power
regulator, all necessary power gates for the device are opened. Closing all
power gates saves about 2.6 Watt.
If we consider all measures taken to save power, we were able to reduce power
consumption to about 59% without performance degradation. When measuring power
consumption after boot up, setting the CPU clock to 1.6 GHz, and fully load
both CPU cores without the described changes, we measured about 8 Watt. With
the described power saving provisions enabled, we measured about 4.7 Watt.
When further reducing the CPU clock frequency to 200 MHz, only 1.7 Watt were
measured.
VESA driver moved to libports
=============================
The VESA framebuffer driver executes the initialization code located
in VESA BIOS of the graphics card. As the BIOS code is for real mode,
the driver uses the x86emu library from X11 as emulation environment.
We updated x86emu to version 1.20 and moved the driver from the 'os'
repository to the 'libports' repository as the library is third-party
code. Therefore, if you want to use the driver, the 'libports'
repository has to be prepared
('make -C <genode-dir>/libports prepare PKG=x86emu') and enabled in
your 'etc/build.conf'.
Runtime environments
####################
Seoul (aka Vancouver) VMM on NOVA
=================================
Since we repeatedly received requests for using the Seoul respectively
Vancouver VMM on NOVA, we improved the support for this virtualization
solution on Genode. Seoul now supports booting from raw hard disk images
provided via Genode's block session interface. Whether this image is actually
a file located in memory, or it is coming directly from the hard disk, or just
from a partition of the hard disk using Genode's part_blk service, is
completely transparent thanks to Genode's architecture.
Additionally, we split up the one large Vancouver run script into several
smaller Seoul run scripts for easier usage - e.g. one for disk, one for
network testing, one for automated testing, and one we call "fancy". The
latter resembles the former vancouver.run script using Genode's GUI to let the
user start VMs interactively. The run scripts prefixed with 'seoul-' can be
found at ports/run. For the fancy and network scripts, ready-to-use VM images
are provided. Those images are downloaded automatically when executing the run
script for the first time.
L4Linux on Fiasco.OC
====================
L4Linux has been updated from version 3.5.0 to Linux kernel version 3.9.0 thus
providing support for contemporary user lands running on top of L4Linux on both
x86 (32bit) and ARM platforms.
Noux runtime for Unix software
==============================
Noux is our way to use the GNU software stack natively on Genode. To improve
its performance, we revisited the address-space management of the runtime to
avoid redundant revocations of memory mappings when Noux processes are cleaned
up.
Furthermore, we complemented the support for the Genode tool chain to
cover GNU sed and GNU grep as well. Both packages are available at the ports
repository.
Platforms
#########
Fiasco.OC updated to revision r56
=================================
Fiasco.OC and the required L4RE parts have been updated to the current SVN
revision (r56). For us, the major new feature is the support of Exynos SOCs in
the mainline version of Fiasco.OC (www.tudos.org). Therefore Genode's
implementation of the Exynos5250 platform could be abandoned leading to less
maintenance overhead of Genode on Fiasco.OC.
Furthermore, Genode's multi-processor support for this kernel has been
improved so that Fiasco.OC users benefit from the additions described in
Section [Enhanced multi-processor support].
NOVA updated
============
In the process of our work on the multi-processor support on NOVA, we updated
the kernel to the current upstream version. Additionally, our customized branch
(called r3) comes with the added cross-CPU IPC system call and improvements
regarding the release of kernel resources.
Integrity checks for downloaded 3rd-party software
##################################################
Even though Genode supports a large variety of 3rd-party software, its
source-code repository contains hardly any 3rd-party source code. Whenever
3rd-party source code is needed, Genode provides automated tools for
downloading the code and integrating it with the Genode environment. As of
now, there exists support for circa 70 software packages, including the
tool chain, various kernels, libraries, drivers, and a few applications. Of
those packages, the code for 13 packages comes directly from their respective
Git repositories. The remaining 57 packages are downloaded in the form of tar
archives from public servers via HTTP or FTP. Whereas we are confident with
the integrity of the code that comes from Git repositories, we are less so
about the archives downloaded from HTTP or FTP servers.
Fortunately, most Open-Source projects provide signature files that allow
the user to verify the origin of the archive. For example, archives of
GNU software are signed with the private key of the GNU project. So the
integrity of the archive can be tested with the corresponding public key.
We used to ignore the signature files for many years but
this has changed now. If there is a signature file available for a package,
the package gets verified right after downloading. If only a hash-sum file
is provided, we check it against a known-good hash sum.
The solution required three steps, the creation of tools for validating
signatures and hashes, the integration of those tools into Genode's
infrastructure for downloading the 3rd-party code, and the definition of
verification rules for the individual packages.
First, new tools for downloading and validating hash sums and signatures were
added in the form of the shell scripts download_hashver (verify hash sum) and
download_sigver (verify signature) found at the tool/ directory. Under the
hood, download_sigver uses GNU GPG, and download_hashver uses the tools
md5sum, sha1sum, and sha256sum provided by coreutils.
Second, hooks for invoking the verification tools were added to the
tool-chain build script as well as the ports and the libports repositories.
The third and the most elaborative step, was going through all the packages,
looking for publicly available signature files, and adding corresponding
package rules. As of now, this manual process has been carried out for 30
packages, thereby covering the half of the archives.
Thanks to Stephan Mueller for pushing us into the right direction, kicking off
the work on this valuable feature, and for the manual labour of revisiting all
the 3rd-party packages!

791
doc/release_notes-14-02.txt
View File

@@ -1,791 +0,0 @@
===============================================
Release notes for the Genode OS Framework 14.02
===============================================
Genode Labs
During the release cycle of version 14.02, our development has been focused on
storage and virtualization. It goes without saying that proper support for
block-device access and file systems is fundamental for the use of
Genode as general-purpose OS. Virtualization is relevant as well because
it bridges the gap between the functionality we need and the features
natively available on Genode today.
Our work on the storage topic involved changes of the block-driver APIs to an
asynchronous mode of operation, overhauling most of the existing block-level
components, as well as the creation of new block services, most importantly a
block cache. At file-system level, we continued our line of work on FUSE-based
file systems, adding support for NTFS-3g. A new highlight, however, is a new
file-system service that makes the file systems of the NetBSD kernel available
to Genode. This is made possible by using rump kernels as described in Section
[NetBSD file systems using rump kernels].
Virtualization on Genode has a long history, starting with the original
support of OKLinux on the OKL4 kernel (OKLinux is no longer supported), over
the support of L4Linux on top of the Fiasco.OC kernel, to the support of the
Vancouver VMM on top of NOVA. However, whereas each of those variants has
different technical merits, all of them were developed in the context of
university research projects and were never exposed to real-world scenarios.
We were longing for a solution that meets the general expectations from a
virtualization product, namely the support for a wide range of guest OSes,
guest-host integration features, ease of use, and an active development.
VirtualBox is one of the most popular commodity virtualization products as of
today. With the current release, we are happy to announce the availability of
VirtualBox on top of Genode/NOVA. Section
[VirtualBox on top of the NOVA microhypervisor] gives insights into the
background of this development, the technical challenges we had to overcome,
and the current state of the implementation.
In addition to addressing storage and virtualization, the current release
comes with a new pseudo file system called trace_fs that allows the
interactive use of Genode's tracing facilities via Unix commands,
a profound unification of the various graphics back ends used throughout
the framework, a new facility for propagating status reports, and
improvements of the Noux runtime for executing Unix software on Genode.
VirtualBox on top of the NOVA microhypervisor
#############################################
Virtualization is an important topic for Genode for two distinct reasons.
It is repeatedly requested by users of the framework who consider
Genode as a microkernel-based hosting platform for virtual machines,
and it provides a smooth migration path from using Linux-based systems
towards using Genode as day-to-day OS.
Why do people consider Genode as a hosting platform for virtual machines
if there is an abundance of mature virtualization solutions on the market?
What all existing popular solutions have in common is the staggering complexity
of their respective trusted-computing base (TCB). The user of a virtual
machine on a commodity hosting platform has to trust millions of lines of
code. For example, with Xen, the TCB comprises the hypervisor and the Linux
system running as DOM0. For security-sensitive application areas, it is
almost painful to trust such a complex foundation. In contrast, the TCB of a
hosting platform based on Genode/NOVA is two orders of magnitude less complex.
Lowering the complexity reduces the likelihood for vulnerabilities and thereby
mitigates the attack surface of the system. It also enables the assessment of
security properties by thorough evaluation or even formal verification. In the
light of the large-scale privacy issues of today, the desire for systems that
are resilient against malware and zero-day exploits has never been higher.
Microkernel-based operating systems promise a solution. Virtualization enables
compatibility to existing software. Combining both seems natural. This is what
Genode/NOVA stands for.
From the perspective of us Genode developers who are in the process of
migrating from Linux-based OSes to Genode as day-to-day OS, we consider
virtualization as a stop-gap solution for all those applications that
do not exist natively on Genode, yet. Virtualization makes our transition
an evolutionary process.
Until now, NOVA was typically accompanied with a co-developed virtual machine
monitor called Seoul (formerly called Vancouver), which is executed as a
regular user-level process on top of NOVA. In contrast to conventional wisdom
about the performance of microkernel-based systems, the Seoul VMM on top of
NOVA is extremely fast, actually faster then most (if not all) commonly used
virtualization solutions. However, originating from a research project, Seoul
is quite challenging to use and not as mature as commodity VMMs that were
developed as real-world products. For example, there is a good chance that an
attempt to boot an arbitrary version of a modern Linux distribution might just
fail. In our experience, it takes a few days to investigate the issues, modify
the guest OS configuration, and tweak the VMM here and there, to run the OS
inside the Seoul VMM. That is certainly not a show stopper in appliance-like
scenarios, but it rules out Seoul as a general solution. Running Windows
OS as guest is not supported at all, which further reduces the application
areas of Seoul. With this in mind, it is unrealistic to propose the use
of Genode/NOVA as an alternative for popular VM hosting solutions.
Out of this realization, the idea was born to combine NOVA's virtualization
interface with a time-tested and fully-featured commodity VMM. Out of the
available Open-Source virtualization solutions, we decided to take a closer
look at VirtualBox, which attracted us for several reasons: First, it is
portable, supporting various host OSes such as Solaris, Windows OS, Linux,
and Mac OS X. Second, it has all the guest-integration features we could
wish for. There are extensive so-called guest additions for popular guest
OSes that vastly improve the guest-OS performance and allow a tight
integration with the host OS using shared folders or a shared clipboard.
Third, it comes with sophisticated device models that support all
important popular guest OSes. And finally, it is actively developed and
commercially supported.
However, moving VirtualBox over to NOVA presented us with a number of
problems. As a precondition, we needed to gain a profound understanding
of the VirtualBox architecture and the code base. To illustrate the challenge,
the source-code distribution of VirtualBox comprises 2.8 million lines of
code. This code contains build tools, the VMM, management tools, several
3rd-party libraries, middleware, the guest additions, and tests. The pieces
that are relevant for the actual VMM amount to 700 thousand lines. By
reviewing the architecture, we found that the part of VirtualBox that
implements the hypervisor functionality (the world switch) runs in the
kernel of the host OS (it is loaded on demand by the user-level VM process
through the _/dev/vboxdrv_ interface into the host OS kernel). It is
appropriately named VMMR0. Once installed into the host OS kernel, it
takes over the control over the machine. To put it blatantly simple, it runs
"underneath" the host OS. The VMMR0 code is kernel agnostic, which explains
the good portability of VirtualBox across various host OSes. Porting
VirtualBox to a new host OS comes down to finding a hook for installing the
VMMR0 code into the host OS kernel and adapting the VirtualBox runtime API
to the new host OS.
In the context of microkernel-based systems, however, it becomes clear that
this classical approach of porting VirtualBox would subvert the microkernel
architecture. Not only would we need to punch a hole into NOVA for loading
additional kernel code, but also the VMMR0 code would inflate the amount of
code executed in privileged mode by more than factor 20. Both implications
are gross violations of the microkernel principle. Consequently, we needed to
find a different way to marry NOVA with VirtualBox.
Our solution was the creation of a drop-in replacement of the VMMR0 code that
runs solely at user level and interacts with NOVA's virtualization
interface. Our VMMR0 emulation code is co-located with the VirtualBox
VM process. Architecturally, the resulting solution is identical to the
use of Seoul on top of NOVA. There is one VM process per virtual machine,
and each VM process is isolated from others by the NOVA kernel. In
addition to creating the VMMR0 emulation code, we needed to replace some parts
of the VirtualBox VMMR3 code with custom implementations because they
overlapped with functionality provided by NOVA's virtualization interface,
in particular the provisioning of guest-physical memory. Finally, we needed
to interface the VM process with Genode's API to let the VM process
interact with Genode's input, file-system, and framebuffer services.
The result of this undertaking is available at the _ports_ repository.
VirtualBox can be downloaded and integrated with Genode via the following
command issued from within the repository:
! make prepare PKG=virtualbox
To illustrate the integration of VirtualBox into a Genode system, there
is run script located at _ports/run/virtualbox.run_. It expects a
bootable ISO image containing a guest OS at _<build-dir>/bin/test.iso_.
The configuration of the VirtualBox process is as simple as
! <config>
! <image type="iso" file="/iso/test.iso" />
! </config>
VirtualBox will try to obtain the specified ISO file via a file-system
session. Furthermore, it will open a framebuffer session and an input session.
The memory assigned to the guest OS depends on the RAM quota assigned to the
VirtualBox process. Booting a guest OS stored in a VDI file is supported. The
image type must be changed to "vdi" accordingly.
Please note that this first version of VirtualBox is far from being complete
as it lacks many features (SMP, guest-addition support, networking), is not
optimized, and must be considered as experimental. However, we could
successfully run GNU/Linux, Android, Windows XP, Windows 7, HelenOS, Minix-3,
GNU Hurd, and of course Genode inside VirtualBox.
One point we are pretty excited about is that the porting effort to
Genode/NOVA did not require any change of Genode. From Genode's point of
view, VirtualBox is just an ordinary leaf node of the process tree, which
can happily co-exist with other processes - even if it is the Seoul VMM.
[image seoul-vbox-win7-tinycore]
In the screenshot above, VirtualBox is running besides the Seoul VMM on top of
Genode/NOVA. Seoul executes Tinycore Linux as guest OS. VirtualBox executes MS
Windows 7. Both VMMs are using hardware virtualization (VT-x) but are plain
user-level programs with no special privileges.
NetBSD file systems using rump kernels
######################################
In the previous release, we made FUSE-based file systems available to Genode
via a custom implementation of the FUSE API. Even though this step made
several popular file systems available, we found that the file systems most
important to us (such as ext) are actually not well supported by FUSE. For
example, write support on ext2 is declared as an experimental feature. In
hindsight it is clear why: FUSE is primarily being used for accessing file
systems not found in the Linux kernel. So it shines with supporting NTFS
but less so with file systems that are well supported by the Linux kernel.
Coincidentally, when we came to this realization, we stumbled upon the
wonderful work of Antti Kantee on so-called rump kernels:
:[http://wiki.netbsd.org/rumpkernel/]:
Rump kernel Wiki
The motivation behind the rump kernels was the development of
NetBSD kernel subsystems (referred to as "drivers") in the NetBSD user land.
Such subsystems like file systems, device drivers, or the TCP/IP stack are
linked against a stripped-down version of the NetBSD kernel that can be
executed in user mode and uses a fairly small "hypercall" interface to
interact with the outside world. A rump kernel contains everything needed to
execute NetBSD kernel subsystems but hardly anything else. In particular, it
does not support the execution of programs on top. From our perspective,
having crafted device-driver environments (DDEs) for Linux, iPXE, and OSS over
the years, a rump kernel sounded pretty much like a DDE for NetBSD. So we
started exploring rump kernels with the immediate goal of making time-tested
NetBSD file systems available to Genode.
To our delight, the integration of rump kernels into the Genode system went
fairly smooth. The most difficult part was the integration of the NetBSD build
infrastructure with Genode's build system. The glue between rump kernels and
Genode is less than 3,000 lines of code. This code enables us to reuse all
NetBSD file systems on Genode. A rump kernel instance that contains several
file systems such as ext2, iso9660, msdos, and ffs takes about 8 MiB of memory
when executed on Genode.
The support for rump kernels comes in the form of the dedicated _dde_rump_
repository. For downloading and integrating the required NetBSD source code,
the repository contains a Makefile providing the usual 'make prepare'
mechanism. To build the file-system server, make sure to add the _dde_rump_
repository to the 'REPOSITORIES' declaration of your _etc/build.conf_ file
within your build directory. The server then can be built via
! make server/rump_fs
There is a run script located at _dde_rump/run/rump_ext2.run_ to execute
a simple test scenario:
! make run/rump_ext2
The server can be configured as follows:
!<start name="rump_fs">
! <resource name="RAM" quantum="8M" />
! <provides><service name="File_system"/></provides>
! <config fs="ext2fs"><policy label="" root="/" writeable="yes"/></config>
!</start>
On startup, it requests a service that provides a block session. If
there is more than one block session in the system, the block session must be
routed to the right block-session server. The value of the _fs_ attribute of
the '<config>' node can be one of the following: _ext2fs_ for EXT2, _cd9660_ for
ISO-9660, or _msdos_ for FAT file-system support. _root_ defines the directory
of the file system as seen as root directory by the client. The server hands
most of its RAM quota to the rump kernel. This means the larger the quota is,
the larger the internal block caches of the rump kernel will be.
Base framework
##############
The base API has not underwent major changes apart from the addition of
a few new utilities and minor refinements. Under the hood, however, the inner
workings of the framework received much attention, including an extensive
unification of the startup code and stack management.
New 'construct_at' utility
==========================
A new utility located at 'base/include/util/construct_at.h' allows for the
manual placement of objects without the need to have a global placement new
operation nor the need for type-specific new operators.
New utility for managing volatile objects
=========================================
Throughout Genode, we maintain a programming style that largely avoids dynamic
memory allocations. For the most part, higher-level objects aggregate
lower-level objects as class members. For example, the nitpicker GUI server
is actually a compound of such aggregations (see
[https://github.com/genodelabs/genode/blob/master/os/src/server/nitpicker/main.cc#L803 - Nitpicker::Main]).
This functional programming style leads to robust programs but it poses a
problem for programs that are expected to adopt their behaviour at runtime.
For the example of nitpicker, the graphics back end of the GUI server takes
the size of the screen as constructor argument. If the screen size changes,
the once constructed graphics back end becomes inconsistent with the new
screen size. We desire a way to selectively replace an aggregated object by a
new version with updated constructor arguments. The new utilities found in
'os/include/util/volatile_object.h' solve this problem. A so-called
'Volatile_object' wraps an object of the type specified as template argument.
In contrast of a regular object, a 'Volatile_object' can be re-constructed any
number of times by calling 'construct' with the constructor arguments. It is
accompanied with a so-called 'Lazy_volatile_object', which remains
unconstructed until 'construct' is called the first time.
Changed interface of 'Signal_rpc_member'
========================================
We unified the 'Signal_rpc_member' interface to be more consistent with the
'Signal_rpc_dispatcher'. The new version takes an entrypoint as argument and
cares for dissolving itself from the entrypoint when destructed.
Filename as default label for ROM connections
=============================================
Since the first version of Genode, ROM services used to rely on a "filename"
provided as session argument. In the meanwhile, we established the use of the
session label to select routing policies as well as server-side policies.
Strictly speaking, the name of a ROM module is used as a key to a server-side
policy of ROM services. So why not to use the session label to express the
key as we do with other services? By assigning the file name as label for ROM
sessions, we may become able to remove the filename argument in the future by
just interpreting the last part of the label as filename. By keeping only the
label, we won't need to consider conditional routing (via '<if-arg>') based on
session arguments other than the label anymore, which would simplify Genode
configurations in the long run. This change is transparent at API level but
may be taken into consideration when configuring Genode systems.
New 'Genode::Deallocator' interface
===================================
By splitting the new 'Genode::Deallocator' interface from the former
'Genode::Allocator' interface, we become able to restrict the accessible
operations for code that is only supposed to release memory, but not
perform any allocations.
Closely related to the allocator interface, we introduced variants of the
'new' operator that take a reference (as opposed to a pointer) to a
'Genode::Allocator' as argument.
Unified main-stack management and startup code among all platforms
==================================================================
In contrast to the stacks of regular threads, which are located within a
dedicated virtual-address region called thread-context area, the stack of
the main thread of a Genode program used to be located within the BSS
segment. If the stack of a normal thread overflows, the program produces
an unresolvable page fault, which can be easily debugged. However,
an overflowing main stack would silently corrupt the BSS segment. With
the current release, we finally resolved this long-standing problem by
moving the main stack to the context area, too. The tricky part was that
the context area is created by the main thread. So we hit a hen-and-egg
problem. We overcame this problem by splitting the process startup
into two stages, both called from the crt0 assembly code. The first
stage runs on a small stack within the BSS and has the sole purpose
of creating the context area and a thread object for the main thread.
This code path (and thereby the stack usage) is the same for all programs.
So we can safely dimension the stage-1 stack. Once the first stage
returns to the crt0 assembly code, the stack pointer is loaded with the
stack that is now located within the context area. Equipped with the
new stack, the actual startup code ('_main') including the global
constructors of the program is executed.
This change paved the ground for several further code unifications and
simplifications, in particular related to the dynamic linker.
Low-level OS infrastructure
###########################
Revised block-driver framework
==============================
Whereas Genode's block-session interface was designed to work asynchronously
and supports the out-of-order processing of requests, those capabilities
remained unused by the existing block services as those services used to
operate synchronously to keep their implementation simple. However, this
simplicity came at the prize of two disadvantages: First, it prevented us
to fully utilize native command queuing of modern disk controllers. Second,
when chaining components such as a block driver, the part_blk server, and
a file system, latencies accumulated along the chain of services. This
hurts the performance of random access patterns.
To overcome this limitation, we changed the block-component framework to work
asynchronously and to facilitate the recently introduced server API.
Consequently, all users of the API underwent an update. The affected
components are rom_loopdev, atapi_drv, fb_block_adapter, http_block, usb_drv,
and part_blk. For some components, in particular part_blk, this step led to a
complete redesign.
Besides the change of the block-component framework, the block-session
interface got extended to support logical block addresses greater than
32bit (LBA48). Thereby, the block component framework can now support
devices that exceed 2 TiB in size.
Block cache
===========
The provisioning of a block cache was one of the primary motivations behind the
[http://www.genode.org/documentation/release-notes/13.11#Dynamic_resource_balancing - dynamic resource balancing]
concept that was introduced in Genode 13.11. We are now introducing the first
version of such a cache.
The new block cache component located at _os/src/server/blk_cache/_ is both
a block-session client as well as a block-session server serving a single
client. It is meant to sit between a block-device driver and a file-system
server. When accessing the block device, it issues requests at a granularity
of 4K and thereby implicitly reads ahead whenever a client requests a smaller
amount of blocks. Blocks obtained from the device or written by the client
are kept in memory. If memory becomes scarce, the block cache first tries
to request further memory resources from its parent. If the request
gets denied, the cache evicts blocks from memory to the block device following
a least-recently-used replacement strategy. As of now, the block cache supports
dynamic resource requests to grow on demand but support for handling yield
requests is not yet implemented. So memory once handed out to the block cache
cannot be regained. Adding support for yielding memory on demand will be
complemented in the next version.
To see how to integrate the block cache in a Genode scenario, there is a
ready-to-use run script available at _os/run/blk_cache.run_.
File-system infrastructure
==========================
In addition to the integration of NetBSD's file systems, there are
file-system-related improvements all over the place.
First, the 'File_system::Session' interface has been extended with a 'sync'
RPC function. This function allows the client of a file system to force
the file system to write back its internal caches.
Second, we extended the FUSE implementation introduced with the previous
release.
Since file systems tend to have a built-in caching mechanism, we need to
sync these caches at the end of a session when using the fuse_fs server.
Therefore, each FUSE file system port has to implement a 'Fuse::sync_fs()'
function that executes the necessary actions if requested. Further
improvements are related to the handling of symbolic links and error
handling. Finally, we added a libc plugin for accessing NTFS file systems
via the ntfs-3g library.
Third, we complemented the family of FUSE-based libc plugins with a family of
FUSE-based file-system servers. To utilize a FUSE file system, there is a
dedicated binary (e.g., _os/src/server/fuse_fs/ext2_) for each FUSE
file-system server.
Note that write support is possible but considered to be experimental at this
point. For now, using it is not recommended.
To use the ext2_fuse_fs server in Noux, the following configuration snippet
may be used:
! <start name="ext2_fuse_fs">
! <resource name="RAM" quantum="8M"/>
! <provides> <service name="File_system"/> </provides>
! <config>
! <policy label="noux -> fuse" root="/" writeable="no" />
! </config>
! </start>
Finally, the libc file-system plugin has been extended to support 'unlink'.
Trace file system
=================
The new _trace_fs_ server provides access to a trace session by providing a
file-system session as front end. Combined with Noux, it allows for the
interactive exploration and tracing of Genode's process tree using
traditional Unix tools.
Each trace subject is represented by a directory ('thread_name.subject') that
contains specific files, which are used to control the tracing process of the
thread as well as storing the content of its trace buffer:
:'enable': The tracing of a thread is activated if there is a valid policy
installed and the intend to trace the subject was made clear by writing '1'
to the 'enable' file. The tracing of a thread may be deactivated by writing a
'0' to this file.
:'policy': A policy may be changed by overwriting the currently used one in the
'policy' file. In this case, the old policy is replaced by the new one and
automatically used by the framework.
:'buffer_size': Writing a value to the 'buffer_size' file changes the size of
the trace buffer. This value is evaluated only when reactivating the tracing
of the thread.
:'events': The trace-buffer contents may be accessed by reading from the
'events' file. New trace events are appended to this file.
:'active': Reading the file will return whether the tracing is active (1) or
not (0).
:'cleanup': Nodes of untraced subjects are kept as long as they do not change
their tracing state to dead. Dead untraced nodes are automatically removed
from the file system. Subjects that were traced before and are now untraced
can be removed by writing '1' to the 'cleanup' file.
To use the trace_fs, a configuration similar to the following may be used:
! <start name="trace_fs">
! <resource name="RAM" quantum="128M"/>
! <provides><service name="File_system"/></provides>
! <config>
! <policy label="noux -> trace"
! interval="1000"
! subject_limit="512"
! trace_quota="64M" />
! </config>
! </start>
:'interval': sets the period the Trace_session is polled. The
time is given in milliseconds.
:'subject_limit': specifies how many trace subjects should by acquired at
max when the Trace_session is polled.
:'trace_quota': is the amount of quota the trace_fs should use for the
Trace_session connection. The remaining amount of RAM quota will be used
for the actual nodes of the file system and the 'policy' as well as the
'events' files.
In addition, there are 'buffer_size' and 'buffer_size_limit' that define
the initial and the upper limit of the size of a trace buffer.
A ready-to-use run script can by found in 'ports/run/noux_trace_fs.run'.
Unified interfaces for graphics
===============================
Genode comes with several programs that perform software-based graphics
operations. A few noteworthy examples are the nitpicker GUI server,
the launchpad, the scout tutorial browser, or the terminal. Most of those
programs were equipped with their custom graphics back end. In some
cases such as the terminal, nitpicker's graphics back end was re-used.
But this back end is severely limited because its sole purpose is the
accommodation of the minimalistic (almost invisible) nitpicker GUI server.
The ongoing work on Genode's new user interface involves the creation of
new components that rely on a graphics back end. Instead of further
diversifying the zoo of graphics back ends, we took the intermediate step
to consolidate the existing back ends into one unified concept such that
application-specific graphics back ends can be created and extended using
modular building blocks. The new versions of nitpicker, scout, launchpad,
liquid_fb, nitlog, and terminal have been changed to use the new common
interfaces:
:os/include/util/geometry.h: Basic data structures and operations needed
for 2D graphics.
:os/include/util/color.h: Common color representation and utilities.
:os/include/os/pixel_rgba.h: Class template for representing a pixel.
:os/include/os/pixel_rgb565.h: Template specializations for RGB565 pixels.
:os/include/os/surface.h: Target surface, onto which graphics operations
can be applied.
:os/include/os/texture.h: Source texture for graphics operations that
transfer 2D pixel data to a surface.
The former _os/include/nitpicker_gfx/_ directory is almost deserted. The only
remainders are functors for the few graphics operations actually required by
nitpicker. For the scout widgets, the corresponding functors have become
available at the public headers at _demo/include/scout_gfx/_.
Because the scout widget set is used by at least three programs and will
most certainly play a role in new GUI components, we undertook a major
cleanup of the parts worth reusing. The result can be found at
_demo/include/scout/_.
New session interface for status reporting
==========================================
Genode has a uniform way of how configuration information is passed from
parents to children within the process tree by the means of "config" ROM
modules. Using this mechanism, a parent is able to steer the behaviour of
its children, not just at their start time but also during runtime.
Until now, however, there was no counterpart to the config mechanism, which
would allow a child to propagate runtime information to its parent. There
are many use cases for such a mechanism. For example, a bus-controller driver
might want to propagate a list of devices attached to the bus. When a new
device gets plugged in, this list should be updated to let the parent
take the new device resource into consideration. Another use case would be the
propagation of status information such as the feature set of a plugin.
Taken to the extreme, a process might expose its entire internal state to its
parent in order to allow the parent to kill and restart the process, and
feed the saved state back to the new process instance.
To cover these use cases, we introduced the new report-session interface. When
a client opens a report session, it transfers a part of its RAM quota to the
report server. In return, the report server hands out a dataspace dimensioned
according to the donated quota. Upon reception of the dataspace, the client
can write its status reports into the dataspace and inform the server about
the update via the 'submit' function. In addition to the mere reporting of
status information, the report-session interface is designed to allow the
server to respond to reports. For example, if the report mechanism is used to
implement a desktop notification facility, the user may interactively respond
to an incoming notification. This response can be reflected to the originator
of the notification via the 'response_sigh' and 'obtain_response' functions.
The new _report_rom_ component is both a report service and a ROM service. It
reflects incoming reports as ROM modules. The ROM modules are named
after the label of the corresponding report session.
Configuration
-------------
The report-ROM server hands out ROM modules only if explicitly permitted by a
configured policy. For example:
! <config>
! <rom>
! <policy label="decorator -> pointer" report="nitpicker -> pointer"/>
! <policy ... />
! ...
! </rom>
! </config>
The label of an incoming ROM session is matched against the 'label' attribute
of all '<policy>' nodes. If the session label matches a policy label, the
client obtains the data from the report client with the label specified in the
'report' attribute. In the example above, the nitpicker GUI server sends
reports about the pointer position to the report-ROM service. Those reports
are handed out to a window decorator (labeled "decorator") as ROM module.
XML generator utility
=====================
With the new report-session interface in place, comes the increased
need to produce XML data. The new XML generator utility located at
_os/include/util/xml_generator.h_ makes this extremely easy, thanks to
C++11 language features. For an example application, refer to
_os/src/test/xml_generator/_ and the corresponding run script at
_os/run/xml_generator.run_.
Dynamic ROM service for automated testing
=========================================
The new _dynamic_rom_ service provides ROM modules that change during the
lifetime of a ROM session according to a timeline. The main purpose of this
service is the automated testing of programs that are able to respond to ROM
module changes, for example configuration changes.
The configuration of the dynamic ROM server contains a '<rom>' sub node per
ROM module provided by the service. Each '<rom>' node hosts a 'name' attribute
and contains a sequence of sub nodes that define the timeline of the ROM
module. The possible sub nodes are:
:'<inline>': The content of the '<inline>' node is assigned to the content
of the ROM module.
:'<sleep>': Sleeps a number of milliseconds as specified via the 'milliseconds'
attribute.
:'<empty>': Removes the ROM module.
At the end of the timeline, it re-starts at the beginning.
Nitpicker GUI server
====================
The nitpicker GUI server has been enhanced to support dynamic screen
resizing. This is needed to let nitpicker respond to screen-resolution
changes, or when using a nested version of nitpicker within a resizable
virtual framebuffer window.
To accommodate Genode's upcoming user-interface concept, we introduced the
notion of a parent-child relationship between nitpicker views. If an existing
view is specified as parent at construction time of a new view, the parent
view's position is taken as the origin of the child view's coordinate space.
This allows for the grouping of views, which can be atomically repositioned by
moving their common parent view. Another use case is the handling of popup
menus in Qt5, which can now be positioned relative to their corresponding
top-level window. The relative position is maintained transparently to Qt when
the top-level window gets repositioned.
Libraries and applications
##########################
Noux runtime for executing Unix software
========================================
Noux plays an increasingly important role for Genode as it allows the use
of the GNU software stack. Even though it already supported a variety of
packages including bash, gcc, binutils, coreutils, make, and vim, some
programs were still limited by Noux' not fully complete POSIX semantics,
in particular with regard to signal handling. For example, it was not
possible to cancel the execution of a long-running process via Control-C.
To overcome those limitations, we enhanced Noux by adding the _kill_ syscall,
reworking the _wait_ and _execve_ syscalls, as well as adding
signal-dispatching code to the Noux libc. Special attention had to be paid to
the preservation of pending signals during the process creation via _fork_ and
_execve_.
The current implementation delivers signals each time a Noux syscall
returns. Signal handlers are executed as part of the normal control flow. This
is in contrast to traditional Unix implementations, which allow the
asynchronous invocation of signal handlers out of band with the regular
program flow. The obvious downside of our solution is that a program that got
stuck in a busy loop (and thereby not issuing any system calls) won't respond
to signals. However, as we regard the Unix interface just as a runtime and not
as the glue that holds the system together, we think that this compromise is
justified to keep the implementation simple and kernel-agnostic. In the worst
case, if a Noux process gets stuck because of such a bug, we certainly can
live with the inconvenience of restarting the corresponding Noux subsystem.
To complement our current activities on the block and file-system levels,
the e2fsprogs-v1.42.9 package as been ported to Noux. To allow the
block-device utilities to operate on Genode's block sessions, we added a new
"block" file system to Noux. Such a block file system can be mounted using a
'<block>' node within the '<fstab>'. By specifying a label attribute, each
block session request can be routed to the proper block session provider:
! <fstab>
! ...
! <dir name="dev">
! <block name="blkdev0" label="block_session_0" />
! </dir>
! ...
! </fstab>
In addition to this file system, support for the DIOCGMEDIASIZE ioctl
request was added. This request is used by FreeBSD and therefore by our
libc to query the size of the block device in bytes.
Qt5 refinements
===============
Our port of Qt5 used to rely on custom versions of synchronization
primitives such as 'QWaitCondition' and 'QMutex'. However, since most of the
usual pthread synchronization functions as relied on by Qt5's regular POSIX
back end have been added to Genode's pthread library by now, we could replace
our custom implementations by Qt5's POSIX version.
Platforms
#########
Execution on bare hardware (base-hw)
====================================
The development of our base-hw kernel platform during this release cycle was
primarily geared towards adding multi-processor support. However, as we
haven't exposed the code to thorough testing yet, we deferred the integration
of this feature for the current release.
We increased the number of usable ARM platforms by adding basic support for
the ODROID XU board.
NOVA microhypervisor
====================
The port of VirtualBox to Genode prompted us to improve the NOVA platform in
the following respects.
NOVA used to omit the propagation of the FPU state of the guest OS to the
virtual machine monitor (VMM) during the world switch between the guest OS and
the VMM. With the Vancouver VMM, which is traditionally used on NOVA, this
omission did not pose any problem because Vancouver would never touch the FPU
state of the guest. So the FPU context of the guest was always preserved
throughout the handling of virtualization events. However, in contrast to the
Vancouver VMM, VirtualBox relies on the propagation of the FPU state between
the guest running in VT-X non-root mode and the guest running within the
VirtualBox recompiler. Without properly propagating the FPU state between both
virtualization back ends, both the guest OS in non-root mode and VirtualBox's
recompiler would corrupt each other's FPU state. After first implementing an
interim solution in our custom version of the kernel, the missing FPU context
propagation had been implemented in the upstream version of NOVA as well.
In contrast to most kernels, NOVA did not allow a thread to yield its current
time slice to another thread. The only way to yield CPU time was to block on
a semaphore or to perform an RPC call. Unfortunately both of those instruments
require the time-receiving threads to explicitly unblock the yielding thread
(by releasing the semaphore or replying to the RPC call). However, there are
situations where the progress of a thread may depend on an external
condition or a side effect produced by another (unknown) thread. One
particular example is the spin lock used to protect (an extremely short)
critical section of Genode's lock metadata. Apparently VirtualBox presented
us with several more use cases for thread-yield semantics. Therefore, we
decided to extend NOVA's kernel interface with a new 'YIELD' opcode to the
'ec_control' system call.

1123
doc/release_notes-14-05.txt
View File

File diff suppressed because it is too large Load Diff

1136
doc/release_notes-14-08.txt
View File

File diff suppressed because it is too large Load Diff

1047
doc/release_notes-14-11.txt
View File

File diff suppressed because it is too large Load Diff

899
doc/release_notes-15-02.txt
View File

@@ -1,899 +0,0 @@
===============================================
Release notes for the Genode OS Framework 15.02
===============================================
Genode Labs
Genode's [http://genode.org/about/road-map - roadmap] for this year puts a
strong emphasis on the consolidation and cultivation of the existing feature
set. With the first release of the year, version 15.02 pays tribute to this
mission by stepping up to extensive and systematic automated testing. As
a precondition for scaling up Genode's test infrastructure, the release
features a highly modular tool kit for exercising system scenarios on a growing zoo
of test machines. Section [Modular tool kit for automated testing] explains
the new tools in detail. In the spirit of improving the existing feature
set, Genode 15.02 vastly improves the performance and stability of our version of
VirtualBox running on the NOVA microhypervisor, solves long-standing shortcomings
of memory management on machines with a lot of RAM, addresses NOVA-related
scalability limitations, stabilizes our Rump-kernel-based file-system server,
and refines the configuration interface of the Intel wireless driver.
As the most significant new feature, the new version introduces virtualization
support for ARM to our custom base-hw kernel. Section [Virtualization on ARM]
outlines the design and implementation of this feature, which was greatly
inspired by NOVA's virtualization architecture and has been developed over the
time span of more than a year.
With respect to platform support, we are happy to accommodate the upcoming
USB-Armory board, which is a computer in the form factor of a USB
stick especially geared towards security applications. Section
[Support for the USB-Armory board] covers the background and the current
state of this line of work.
Virtualization on ARM
#####################
The ARMv7 architecture of recent processors like Cortex-A7, Cortex-A15, or
Cortex-A17 CPUs support hardware extensions to facilitate virtualization of
guest operating systems. With the current release, we enable the use of these
virtualization extensions in our custom base-hw kernel when running on the
Cortex-A15-based Arndale board.
While integrating ARM's virtualization extension, we aimed to strictly follow
microkernel-construction principles. The primary design is inspired by the
[http://hypervisor.org/ - NOVA OS Virtualization Architecture]. It is based on a
microhypervisor that provides essential microkernel mechanisms along with
basic primitives to switch between virtual machines (VMs). On top of the
microhypervisor, classical OS services are implemented as
ordinary, unprivileged user-level components. Those services can be used by other
applications. Services may be shared between applications or instantiated
separately, according to security and safety needs. Correspondingly,
following the NOVA principles, each VM has its own associated virtual-machine
monitor (VMM) that runs as an unprivileged user-level component. VMM implementations
can range from simple ones that just emulate primary device requirements to highly
complex monitors including sophisticated device models, like VirtualBox. The
NOVA approach allows to decouple the TCB complexity of one VM with respect to
another, as well as with respect to all components not related to
virtualization at all.
Along those lines, we extended the base-hw kernel/core conglomerate with API
extensions that enable user-level VMM components to create and control virtual
machines.
Design
======
The ARM virtualization extensions are based on the so-called security
extensions, commonly known as
[http://genode.org/documentation/articles/trustzone - TrustZone].
The ARM designers did not follow the
Intel approach to split the CPU into a "root" and a "guest" world while having all prior
existing CPU modes available in both worlds. Instead, ARM added a new privilege level
to the non-secure side of TrustZone that sits underneath the ordinary kernel
and userland privilege levels. It is subjected to a hypervisor-like kernel. All
instructions used to prepare a VM's environment have to be executed in this so
called "hyp" mode. In hyp mode, some instructions
differ from their regular behaviour on the kernel-privilege level.
For this reason, prior-existing kernel code cannot simply be reused in
hyp mode without modifications.
The base-hw kernel is meant to execute Genode's core component on bare hardware.
Core, which is an ordinary user-level component, is
linked together with a slim kernel library that is executed in privileged kernel
mode. To enable ARM hardware virtualization, we pushed this approach
even further by executing core in three different privilege levels. Thereby,
core shares the same view on hardware resources and virtual memory across all
levels. A code path is executed on a higher privilege level only if the code
would fail to execute on a lower privilege level.
Following this approach, we were able to keep most of the existing kernel code
with no modifications.
[image avirt_overview]
Genode's ARM kernel (core) runs across all privilege levels
The hypervisor part of core is solely responsible to switch between VMs and the
host system. Therefore, it needs to load/store additional CPU state that
normally remains untouched during context switches of ordinary tasks. It also needs to
configure the VM's guest-physical to host-physical memory translations. Moreover, the
virtualization extensions of the ARMv7 architecture are not related to the CPU
cores only. The interrupt controller and the CPU-local timers are also
virtualization-aware. Therefore, the hypervisor has to load/store state specific
to those devices, too. Nevertheless, the hypervisor merely reloads those
devices. It does not interpret their state.
In contrast to the low-complexity hypervisor, a user-level VMM can be complex
without putting the system's security at risk. It contains potentially complex
device-emulation code and assigns hardware resources such as memory and
interrupts to the VM. The VMM is an ordinary user-level component running
unprivileged. Of course, as a plain user-level component, it is not able to
directly access hardware resources. Hence an interface between VMMs and the
kernel is needed to share the state of a virtual machine. In the past, we faced a similar
problem when building a VMM for our former TrustZone experiments. It was natural
to build upon the available solution and to extend it where necessary. Core
provides a so-called VM service. Each VM corresponds to a session of this
service. The session provides the following extended interface:
:CPU state:
The CPU-state function returns a dataspace containing the virtual machine's
state. The state is initialized by the VMM before bootstrapping the VM, gets updated
by the hypervisor whenever it switches away from the VM, and can be used by
the VMM to interpret the behavior of the guest OS. Moreover, the CPU state can be
updated after the virtual machine monitor emulated instructions
for the VM.
:Exception handler:
The second function is used to register a signal handler that gets informed
whenever the VM produces a virtualization fault.
:Run:
The run function starts or resumes the execution of the VM.
:Pause:
The pause function removes the VM from the kernel's scheduler.
:Attach:
This function attaches a given RAM dataspace to a designated area of the
guest-physical address space.
:Detach:
The detach function invalidates a designated area of the guest-physical
address space.
:Attach_pic: Tells the hypervisor to attach the CPU's virtual interface of the
virtualization-aware interrupt controller to a designated area of the
guest-physical address space.
Implementation
==============
By strictly following the micro-kernel construction principles when integrating the
hypervisor into the base-hw kernel, we reached a minimally invasive solution. In
doing so, we took the time to separate TrustZone-specific code that was formerly
an inherent part of the kernel on ARMv7 platforms. Now, TrustZone- and
virtualization-specific aspects are incorporated into the kernel only if
actually used. The change in complexity of the whole core component expressed in
lines of code is shown in the table below. As can be seen, the additional code in
the root of the trusted computing base when using virtualization is about 700-800
LOC.
Platform | with TrustZone, no VT | TrustZone/VT optional
-----------------------------------------------------------------
hw_arndale | 17970 LOC | 18730 LOC
----------------------------------------------------------------
hw_imx53_qsb | 17900 LOC | 17760 LOC
----------------------------------------------------------------
hw_imx53_qsb_tz | 18260 LOC | 18320 LOC
----------------------------------------------------------------
hw_rpi | 17500 LOC | 17430 LOC
----------------------------------------------------------------
hw_panda | 18040 LOC | 17880 LOC
----------------------------------------------------------------
hw_odroid_xu | 17980 LOC | 18050 LOC
Besides the VM world switch, we enabled support for the so-called "large
physical address extension" (LPAE), which is obligatory when using
virtualization. It allows for addressing a 40-bit instead of only 32-bit physical
address space. Moreover, to execute in hypervisor mode, the bootstrap code of
the kernel had to be set up properly. Hence, when booting on the Arndale board,
the kernel now prepares the non-secure TrustZone world first, and finally leaves the
secure world forever.
To test and showcase the ARM virtualization features integrated in base-hw, we
implemented a minimal, exemplary VMM. It can be found in
_repos/os/src/server/vmm_. The VMM emulates a simplified variant of ARM's
Versatile Express Cortex-A15 development platform. Currently, it only comprises
support for the CPU, the timer, the interrupt controller, and a UART device. It is
written in 1100 lines of C++ in addition to the base Genode libraries. The VMM
is able to boot a vanilla Linux kernel compiled with a slightly modified
standard configuration (no-SMP), and a device tree description stripped down to
the devices provided by the VMM. This release includes an automated run test that
executes the Linux kernel on top of the VMM on Genode. It can be started via:
! make run/vmm
[image avirt_screen]
Three Linux serial consoles running in parallel on top of Genode
Modular tool kit for automated testing
######################################
In
[http://genode.org/documentation/release-notes/13.05#Automated_quality-assurance_testing - Genode version 13.05],
we already introduced comprehensive support for the automated testing of
Genode scenarios. Since then, Genode Labs has significantly widened the scope
of its internal test infrastructure, both in terms of the coverage of the test
scenarios as well as the variety of the used hardware platforms.
The centerpiece of our test infrastructure is the so-called run tool. Steered
by a script (run script), it performs all the steps necessary to test drive
a Genode system scenario. Those steps are:
# *Building* the components of a scenario
# *Configuration* of the init component
# Assembly of the *boot directory*
# Creation of the *boot image*
# *Powering-on* the test machine
# *Loading* of the boot image
# Capturing the *LOG output*
# *Validation* of the scenario behavior
# *Powering-off* the test machine
Each of those steps depends on various parameters such as the
used kernel, the hardware platform used to run the scenario, the
way the test hardware is connected to the test infrastructure
(e.g., UART, AMT, JTAG, network), the way the test hardware is powered or
reseted, or the way of how the scenario is loaded into the test hardware.
Naturally, to accommodate the growing variety of combinations of those
parameters, the complexity of the run tool increased over time.
This growth of complexity prompted us to eventually turn the run tool into a
highly modular and extensible tool kit.
Originally, the run tool consisted of built-in rules that could be
extended and tweaked by a kernel-specific supplement called run environment.
The execution of a run script used to depend on the policies built into
the run tool, the used run environment, and optional configuration
parameters (run opts).
The new run tool kit replaces most of the formerly built-in policies by the
ability to select and configure different modules for the various steps.
The selection and configuration of the modules is expressed in the run-tool
configuration. There exist the following types of modules:
:boot-dir modules:
These modules contain the functionality to populate the boot directory
and are specific to each kernel. It is mandatory to always include the
module corresponding to the used kernel.
_(the available modules are: linux, hw, okl4, fiasco, pistachio, nova,_
_codezero, foc)_
:image modules:
These modules are used to wrap up all components used by the run script
in a specific format and thereby prepare them for execution.
Depending on the used kernel, different formats can be used. With these
modules, the creation of ISO and disk images is also handled.
_(the available modules are: uboot, disk, iso)_
:load modules:
These modules handle the way the components are transfered to the
target system. Depending on the used kernel there are various options
to pass on the components. For example, loading from TFTP or via JTAG is handled
by the modules of this category.
_(the available modules are: tftp, jtag, fastboot)_
:log modules:
These modules handle how the output of a currently executed run script
is captured.
_(the available modules are: qemu, linux, serial, amt)_
:power_on modules:
These modules are used for bringing the target system into a defined
state, e.g., by starting or rebooting the system.
_(the available modules are: qemu, linux, softreset, powerplug, amt)_
:power_off modules:
These modules are used for turning the target system off after the
execution of a run script.
_(the available modules are: powerplug)_
When executing a run script, only one module of each category must be used.
Each module has the form of a script snippet located under the
_tool/run/<step>/_
directory where _<step>_ is a subdirectory named after the module type.
Further instructions about the use of each module (e.g., additional
configuration arguments) can be found in the form of comments inside the
respective script snippets.
Thanks to this modular structure,
the extension of the tool kit comes down to adding a file at the corresponding
module-type subdirectory. This way, custom work flows (such as tunneling JTAG
over SSH) can be accommodated fairly easily.
Usage examples
==============
To execute a run script, a combination of modules may be used. The combination
is controlled via the RUN_OPT variable used by the build framework. Here are a
few common exemplary combinations:
Executing NOVA in Qemu:
!RUN_OPT = --include boot_dir/nova \
! --include power_on/qemu --include log/qemu --include image/iso
Executing NOVA on a real x86 machine using AMT for resetting the target system
and for capturing the serial output while loading the files via TFTP:
!RUN_OPT = --include boot_dir/nova \
! --include power_on/amt --power-on-amt-host 10.23.42.13 \
! --power-on-amt-password 'foo!' \
! --include load/tftp --load-tftp-base-dir /var/lib/tftpboot \
! --load-tftp-offset-dir /x86 \
! --include log/amt --log-amt-host 10.23.42.13 \
! --log-amt-password 'foo!'
Executing Fiasco.OC on a real x86 machine using AMT for resetting, USB serial
for output while loading the files via TFTP:
!RUN_OPT = --include boot_dir/foc \
! --include power_on/amt --amt-host 10.23.42.13 --amt-password 'foo!' \
! --include load/tftp --tftp-base-dir /var/lib/tftpboot \
! --tftp-offset-dir /x86 \
! --include log/serial --log-serial-cmd 'picocom -b 115200 /dev/ttyUSB0'
Executing base-hw on a Raspberry Pi using powerplug to reset the hardware,
JTAG to load the image and USB serial to capture the output:
!RUN_OPT = --include boot_dir/hw \
! --include power_on/powerplug --power-on-powerplug-ip 10.23.42.5 \
! --power-on-powerplug-user admin \
! --power-on-powerplug-password secret \
! --power-on-powerplug-port 1
! --include power_off/powerplug --power-off-powerplug-ip 10.23.42.5 \
! --power-off-powerplug-user admin \
! --power-off-powerplug-password secret \
! --power-off-powerplug-port 1
! --include load/jtag \
! --load-jtag-debugger /usr/share/openocd/scripts/interface/flyswatter2.cfg \
! --load-jtag-board /usr/share/openocd/scripts/interface/raspberrypi.cfg \
! --include log/serial --log-serial-cmd 'picocom -b 115200 /dev/ttyUSB0'
After the run script was executed successfully, the run tool will print the
string 'Run script execution successful.". This message can be used to check
for the successful completion of the run script when doing automated testing.
Meaningful default behaviour
============================
To maintain the ease of use of creating and using a build directory, the
'create_builddir' tool equips a freshly created build directory with a meaningful
default configuration that depends on the selected platform. For example, if
creating a build directory for the Linux base platform, RUN_OPT
is initially defined as
! RUN_OPT = --include boot_dir/linux \
! --include power_on/linux --include log/linux
Low-level OS infrastructure
###########################
Improved management of physical memory
======================================
On machines with a lot of memory, there exist constraints with regard to
the physical address ranges of memory:
* On platforms with a non-uniform memory architecture, subsystems should
preferably use memory that is local to the CPU cores the subsystem is using.
Otherwise the performance is impeded by costly memory accesses to
the memory of remote computing nodes.
* Unless an IOMMU is used, device drivers program physical addresses
into device registers to perform DMA operations. Legacy devices such as
USB UHCI controllers expect a 32-bit address. Consequently, the memory
used as DMA buffers for those devices must not be allocated above 4 GiB.
* When using an IOMMU on NOVA, Genode represents the address space
accessible by devices (by the means of DMA) using a so-called device PD
([http://genode.org/documentation/release-notes/13.02#DMA_protection_via_IOMMU]).
DMA transactions originating from PCI devices are subjected to the virtual
address space of the device PD.
All DMA buffers are identity-mapped with their physical addresses within
the device PD. On 32-bit systems with more than 3 GiB of memory, this
creates a problem. Because the device PD is a regular user-level component, the
upper 1 GiB of its virtual address space is preserved for the kernel. Since
no user-level memory objects can be attached to this
area, the physical address range to be used for DMA buffers is limited
to the lower 3 GiB.
Up to now, Genode components had no way to influence the allocation of
memory with respect to physical address ranges. To solve the problems outlined
above, we extended core's RAM services to take allocation constraints
as session arguments when a RAM session is created. All dataspaces created
from such a session are subjected to the specified constraints. In particular,
this change enables the AHCI/PCI driver to allocate DMA buffers at suitable
physical address ranges.
This innocent looking feature to constrain RAM allocations raises a problem
though: If any component is able to constrain RAM allocations in
arbitrary ways, it would become able to scan the physical address space for
allocated memory by successively opening RAM sessions with the constraints set
to an individual page and observe whether an allocation succeeds or not. Two
conspiring components could use this information to construct a covert storage
channel.
To prevent such an abuse, the init component filters out allocations
constrains from RAM-session requests unless explicitly permitted. The
permission is granted by supplementing the RAM resource assignment of
a component with a new 'constrain_phys' attribute. For example:
! <resource name="RAM" quantum="3M" constrain_phys="yes"/>
Init component
==============
Most of Genode's example scenarios in the form of run scripts support
different platforms. However, as the platform details vary, the run scripts
have to tweak the configuration of the init component according to the
features of the platform.
For example, when declaring an explicit route to a framebuffer driver named
"fb_drv", the run script won't work on Linux because on this platform, the
framebuffer driver is called "fb_sdl".
Another example is the role of the USB driver. Depending on the platform, the
USB driver is an input driver, a block driver, a networking driver, or a
combination of those.
Consequently, run scripts with support
for a great variety of platforms tend to become convoluted with
platform-specific conditionals.
To counter this problem, we enhanced init to support aliases for component
names. By defining the following aliases in the init configuration
! <alias name="nic_drv" child="usb_drv"/>
! <alias name="input_drv" child="usb_drv"/>
! <alias name="block_drv" child="usb_drv"/>
the USB driver becomes reachable for session requests routed to either "usb_drv",
"nic_drv", "input_drv", and "block_drv". Consequently, the routing
configuration of components that use either of those drivers does no longer
depend on any platform-intrinsic knowledge.
RTC session interface
=====================
Until now, the RTC session interface used an integer to return the current
time. Although this is preferable when performing time-related
calculations, a structured representation is more convenient to use, i.e., if
the whole purpose is showing the current time. This interface change is only
visible to components that use the RTC session directly.
Since the current OS API of Genode lacks time-related functions, most users
end up using the libc, which already converts the structured time stamp
internally, or provide their own time related functions.
Update of rump-kernel-based file systems
========================================
We updated the rump-kernel support to a newer rump-kernel version (as of mid of
January 2015). This way, Genode is able to benefit from upstream stability
improvements related to the memory management. Furthermore, we revised the
Genode backend to allow the rump_fs server to cope well with a large amount of
memory assigned to it. The latter is useful to utilize the block cache of the
NetBSD kernel.
Libraries and applications
##########################
As a stepping stone in the
[https://github.com/genodelabs/genode/issues/1399 - forthcoming community effort]
to bring the Nix package manager to Genode, ports of libbz2 and sqlite have
been added to the _repos/libports/_ repository.
Runtime environments
####################
VirtualBox on NOVA
==================
Whereas our previous efforts to run VirtualBox on Genode/NOVA were mostly
concerned with enabling principal functionality and with the addition of
features, we took the release cycle of Genode 15.02 as a chance to focus
on performance and stability improvements.
:Performance:
Our goal with VirtualBox on NOVA is to achieve a user experience
comparable to running VirtualBox on Linux. Our initial port of VirtualBox used
to cut a lot of corners with regards to performance and timing accuracy
because we had to concentrate on more fundamental issues of the porting
work first. Now, with the feature set settled, it was time to revisit
and solidify our interim solutions.
The first category of performance improvements is the handling of timing,
and virtual guest time in particular. In our original version,
we could observe a substantial drift of the guest time compared to the host time.
The drift is not merely inconvenient but may even irritate the guest OS
because it violates its assumptions about the behaviour of certain virtual devices.
The drift was caused by basing the timing on a simple jiffies counter
that was incremented by a thread after sleeping for a fixed period. Even
though the thread almost never executes, there is still a chance that it gets
preempted by the kernel and resumed only after the time slices of
concurrently running threads have elapsed. This can take tens of milliseconds.
During this time, the jiffies counter remains unchanged. We could
significantly reduce the drift by basing the timing on absolute time values
requested from the timer driver. Depending on the used guest OS, however,
there is still a residual inaccuracy left, which is subject to ongoing
investigations.
The second type of improvements is related to the handling of virtual
interrupts. In its original habitat, VirtualBox relies on so-called
external-interrupt virtualization events. If a host interrupt occurs while the
virtual machine is active, the virtualization event is forwarded by the
VirtualBox hypervisor to the virtual machine monitor (VMM).
On NOVA, however, the kernel does not propagate this
condition to the user-level VMM because the occurrence of host interrupts should
be of no matter to the VMM. In the event of a host interrupt, NOVA takes
a normal scheduling decision (eventually activating the user-level device driver
the interrupt belongs to) and leaves the virtual CPU (vCPU) in a runnable
state - to be rescheduled later. Once the interrupt is handled, the vCPU gets
resumed. The VMM remains out of the loop. Because the update of the VirtualBox
device models ultimately relies on the delivery of external-interrupt
virtualization events, the lack of this kind of event introduced huge delays
with respect to the update of device models and the injection of virtual
interrupts. We solved this problem by exploiting a VirtualBox-internal
mechanism called POKE. By setting the so-called POKE flag, an I/O thread is
able to express its wish to force the virtual machine into the VMM. We only
needed to find the right spots to set the POKE flag.
Another performance-related optimization is the caching of RTC time
information inside VirtualBox. The original version of the gettimeofday
function used by VirtualBox contacted the RTC server for obtaining the
wall-clock time on each call. After the update to VirtualBox 4.3, the rate of those
calls increased significantly. To reduce the costs of these calls, our
new version of gettimeofday combines infrequent calls to the RTC driver
with a component-local time source based on the jiffies mechanism mentioned above.
With these optimizations in place,
simple benchmarks like measuring the boot time of Window 7 or the time of
compiling Genode within a Debian VM suggest that our version of VirtualBox
has reached a performance that is roughly on par with the Linux version.
:Stability:
Since the upgrade to VirtualBox 4.3.16 in release 14.11, we fixed several
regression issues caused by the upgrade. Beside that, we completed the
support to route serial output of guests to Genode, lifted the restriction
to use just one fixed VESA mode, and enabled support for 32-bit Windows 8
guests on 64-bit Genode/NOVA. The 64-bit host restriction stems from
the fact that Windows 8 requires support for the non-executable bit (NX)
feature of page tables. The 32-bit version of the NOVA kernel does not leverage
the physical address extension (PAE) feature, which is a pre-requisite for
using NX on 32-bit.
In the course of the adaptation, our port of VirtualBox now evaluates the
PAE and HardwareVirtExUX XML tags of .vbox files:
!<VirtualBox xmlns=...>
! <Machine uuid=...>
! <Hardware ..>
! <CPU ...>
! <HardwareVirtExUX enabled="true"/>
! <PAE enabled="true"/>
! ...
The PAE tag specifies whether to report PAE capabilities to the guest
or not. The HardwareVirtExUx tag is used by our port to decide whether to stay
for non-paged x86 modes in Virtualbox's recompiler (REM) or not. Until now, we used REM
to emulate execution when the guest was running in real mode and protected mode
with paging disabled. However, newer Intel machines support the unrestricted guest
feature, which makes the usage of REM in non-paged modes not strictly
necessary anymore. Setting the HardwareVirtExUx tag to false accommodates
older machines with no support for the unrestricted-guest feature.
Device drivers
##############
iPXE-based network drivers
==========================
We enabled and tested the driver with Intel I218-LM and I218-V PCI devices.
Intel wireless stack
====================
In this release, several small issues regarding the wireless stack are fixed.
From now on, the driver only probes devices on the PCI bus that correspond to
the PCI_CLASS_NETWORK_OTHER device class. Prior to that, the driver probed all
devices attached to the bus resulting in problems with other devices, e.g.
the GPU, when accessing their extended PCI config space.
Since the driver uses cooperative scheduling internally, it must never block
or, in case it blocks, must schedule another task. Various sleep functions
lacked this scheduling call and are now fixed. Furthermore, a bug in the timer
implementation has been corrected, which caused the scheduling of wrong timeouts.
In addition to these fixes, patches for enabling the support for
Intel 7260 cards were incorporated.
Up to now, the configuration of the wireless driver was rather inconvenient because
it did not export any information to the system. The driver now creates two
distinct reports to communicate its state and information about the wireless
infrastructure to other components. The first one is a list of all available
access points. The following exemplary report shows its structure:
!<wlan_accesspoints>
! <accesspoint ssid="skynet" bssid="00:01:02:03:04:05" quality="40"/>
! <accesspoint ssid="foobar" bssid="01:02:03:04:05:06" quality="70" protection="WPA-PSK"/>
! <accesspoint ssid="foobar" bssid="01:02:03:04:05:07" quality="10" protection="WPA-PSK"/>
!</wlan_accesspoints>
Each '<accesspoint>' node has attributes that contain the SSID and the BSSID
of the access point as well as the link quality (signal strength). These
attributes are mandatory. If the network is protected, the node will also
have an attribute describing the type of protection in addition.
The second report provides information about the state of the connection
with the currently associated access point:
!<wlan_state>
! <accesspoint ssid="foobar" bssid="01:02:03:04:05:06" quality="70"
! protection="WPA-PSK" state="connected"/>
!</wlan_state>
Valid state values are 'connected', 'disconnected', 'connecting' and
'disconnecting'.
The driver obtains its configuration via a ROM module. This ROM
module contains the selected access point and can be updated during runtime.
To connect to an access point, a configuration like the following is used:
!<selected_accesspoint ssid="foobar" bssid="01:02:03:04:05:06"
! protection="WPA-PSK" psk="foobar123!"/>
To disconnect from an access point, an empty configuration can be set:
!<selected_accesspoint/>
For now, the prevalent WPA/WPA2 protection using a pre-shared key is supported.
Improved UART driver for Exynos5
================================
The UART driver for the Exynos5 SoC has been enhanced by enabling the RX
channel. This improvement was motivated by automated tests, where a run script
needs to interact with some component via a terminal connection.
Touchscreen support
===================
We enabled support of Wacom USB touchscreen devices via dde_linux - a port of
Linux USB driver to Genode. In order to make touchscreen coordinates
usable by Genode's input services, they must be calibrated
to screen-absolute coordinates. The screen resolution is not determined
automatically by the USB driver. It can, however, be configured as a sub
node of the '<hid>' XML tag of the USB driver's configuration:
!<start name="usb_drv">
! ...
! <config uhci=... ohci=... xhci=...>
! <hid>
! <screen width="1024" height="768"/>
! </hid>
! ...
USB session interface
=====================
We enhanced our USB driver with the support of remote USB sessions. This
feature makes it possible to implement USB-device drivers outside the USB
server using a native Genode API. The new USB session can be found under
_repos/os/include/usb_session_ and can be used to communicate with the USB
server, which merely acts as a host controller and HUB driver in this scenario.
Under _repos/os/include/usb_, there are a number of convenience
and wrapper functions that operate directly on top of a USB session. These
functions are meant to ease the task of USB-device-driver programming by hiding
most of the USB session management, like packet-stream handling.
We also added a USB terminal server, which exposes a Genode terminal session to
its clients and drives the popular PL2303 USB to UART adapters using the new
USB-session interface.
A practical use case for this component is the transmission of logging data on
systems where neither UART, AMT, nor JTAG are available. A run script
showcasing this feature can be found at _repos/dde_linux/run/usb_terminal.run_.
RTC proxy driver for Linux
==========================
There are a handful of run scripts that depend on the RTC service. So far,
it was not possible to run these tests on Linux due to the lack of an RTC
driver on this platform. To address this problem, we created a proxy driver
that uses the time() system call to provide a
reasonable base period on Linux.
Platforms
#########
Execution on bare hardware (base-hw)
====================================
Support for the USB-Armory board
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
With [https://www.crowdsupply.com/inverse-path/usb-armory - USB Armory],
there is an intriguing hardware platform for Genode on the horizon.
In short, USB Armory is a computer in the form factor of a USB
stick. It is meant for security applications such as VPNs,
authentication tokens, and encrypted storage. It is based on the
FreeScale i.MX53 SoC, which is well supported by Genode, i.e.,
Genode can be used as secure-world OS besides Linux running in the
normal world.
Apart from introducing a novel form factor, this project is
interesting because it strives to be an 100% open platform, which
includes hardware, software, and firmware. This motivated us to
bring Genode to this platform.
The underlying idea is to facilitate
[http://genode.org/documentation/articles/trustzone - ARM TrustZone] to
use Genode as a companion to a Linux-based OS on the platform.
Whereas Linux would run in the normal world of TrustZone, Genode runs
in the secure world. With Linux, the normal world will control the
communication over USB and provide a familiar environment to implement
USB-Armory applications. However, security-critical functions and data like
cryptographic keys will reside exclusively in the secure world. Even in
the event that Linux gets compromised, the credentials of the user
will stay protected.
The support of the USB Armory platform was added in two steps:
First, we enabled our base-hw kernel to run as TrustZone monitor with
Genode on the "secure side". Since the USB Armory is based on the
FreeScale i.MX53 SoC, which Genode already supported, this step went
relatively straight-forward.
Second, we enabled a recent version of the Linux kernel (3.18) to run in the
normal world. The normal world is supervised by a user-level Genode component
called tz_vmm (TrustZone Virtual Machine Monitor). The tz_vmm is, among
others, responsible for providing startup and hardware information to the
non-secure guest. The Linux kernel version we used previously as TrustZone
guest on i.MX53 boards expected this information to be communicated via
so-called ATAGs. The new version, however, expects this to be done via a
device tree blob. As a consequence, the tz_vmm had to be adapted to properly
load this blob into the non-secure RAM. The original USB-Armory device tree
was modified to blind out the RAM regions that get protected by the TrustZone
hardware. This way, Linux won't attempt to access them. Furthermore,
to keep basic user interaction simple, our device tree tells Linux to use the
same non-secure UART as Genode for console I/O.
The kernel itself received some modifications, for two reasons. First,
we don't want Linux to rely on resources that are protected to keep
the secure world secure. This is why the driver for the interrupt controller
that originally made use of the TrustZone interrupt configuration, had to be
adapted. Second, to prevent Linux from disturbing Genode activities, we
disabled most of the dynamic clock and power management as it may sporadically
gear down or even disable hardware that Genode relies on. Furthermore, we
disabled the Linux drivers for I2C interfaces and the GPIO configuration as
these are reserved for Genode.
IPC helping
~~~~~~~~~~~
In traditional L4 microkernels, scheduling parameters (like time-slice
length and priority) used to be bound to threads. Usually, those parameters
are defined at thread creation time. The initial version
of base-hw followed this traditional approach. However, it has a few problems:
* For most threads, the proper *choice of scheduling parameters* is very
difficult if not impossible. For example, the CPU-time demands of a
server thread may depend on the usage patterns of its clients. Most
theoretical work in the domain of scheduling presumes the knowledge of
job lengths in advance of computing a schedule. But in practice and in
particular in general-purpose computing, job lengths are hardly known a priori.
As a consequence, in most scenarios, scheduling parameters are
set to default values.
* With each thread being represented as an independent schedulable entity,
the kernel has to take a scheduling decision each time a thread performs an
IPC call because the calling thread gets blocked and the called thread
may get unblocked. In a microkernel-based system, those events occur at a
much higher rate than the duration of typical time slices, which puts the
scheduler in a *performance-critical* position.
* Regarding IPC calls, a synchronous flow of control along IPC call chains is
desired. Ideally, an IPC call should have the same characteristics as
a function call with respect to scheduling. When a client thread performs an
IPC call, it expects the server to immediately become active to
handle the request. But if the kernel treats each thread independently,
it may pick any other thread and thereby introduce *high latencies* into
IPC operations.
To counter those problems, the NOVA microhypervisor introduced a new approach
that decouples scheduling parameters from threads. Instead of selecting
threads for execution, the scheduler selects so-called scheduling contexts.
For a selected scheduling context, the kernel dynamically determines a
thread to execute by taking IPC relationships into account. When a thread
performs an IPC, the thread's scheduling context will be used to execute
the called server. In principle, a server does not need CPU time on its own
but always works with CPU resources provided by clients.
The new version of the base-hw kernel adapts NOVA's approach with slight
modifications. Each thread owns exactly one scheduling context for its entire
lifetime. However, by the means of "helping" during an IPC call, the caller
lends its scheduling context to the callee. Even if the callee is still busy
and cannot handle the IPC request right away, the caller helps because it
wants the callee to become available for its request as soon as
possible. Consequently, a thread has potentially many scheduling contexts at
its disposal, its own scheduling context plus all scheduling contexts
provisioned by helpers. This works transitively.
Purged outdated platforms
~~~~~~~~~~~~~~~~~~~~~~~~~
We removed the support for two stale platforms that remained unused for
more than a year, namely FreeScale i.MX31 and the TrustZone variant
of the Coretile Versatile Express board.
NOVA
====
On Genode/NOVA, we used to employ one pager thread in core for each thread
in the system. We were forced to do so because not every page
fault can be resolved immediately. In some situations, core asynchronously
propagates the fault to an external component for the resolution.
In the meantime, the
pager thread leaves the page fault unanswered. Unfortunately, the kernel
provides no mechanism to support this scenario besides just blocking the
pager thread using a semaphore. This, in turn, means that the pager thread is not
available for other page-fault requests. Ultimately, we had to setup a
dedicated pager per thread.
This implementation has the downside of "wasting" memory for a lot of
pager threads. Moreover, it becomes a denial-of-service vector as soon as more
threads get created than core can accommodate. The number of threads is
limited per address space - also for core - by the size of Genode's context
area, which typically means 256 threads.
To avoid the downsides mentioned, we extended the NOVA IPC reply syscall to
specify an optional semaphore capability. The NOVA kernel validates the
capability and blocks the faulting thread in the semaphore. The faulted thread
remains blocked even after the pager has replied to the fault message. But
the pager immediately becomes available for other
page-fault requests. With this change, it suffices to maintain only one pager
thread per CPU for all client threads.
The benefits are manifold. First, the base-nova implementation converges more
closely to other Genode base platforms. Second, core can not run out of threads
anymore as the number of threads in core is fixed for a given setup. And the
third benefit is that the helping mechanism of NOVA can be leveraged for
concurrently faulting threads.
Build system and tools
######################
Tools for convenient handling of port contrib directories
=========================================================
We supplemented our tools for the ports mechanism with two convenient
scripts:
:_tool/ports/shortcut_:
Creates a symbolic link from _contrib/<port-name>-<hash>_ to
_contrib/<port-name>_. This is useful when working on the third-party
code contained in the _contrib_ directory.
:_tool/ports/current_:
Prints the current contrib directory of a port. When switching
branches back and forth, the hash of the used port might change.
The script provides a shortcut to looking up the hash file for a
specific port within the repositories and printing its content.

1216
doc/release_notes-15-05.txt
View File

File diff suppressed because it is too large Load Diff

791
doc/release_notes-15-08.txt
View File

@@ -1,791 +0,0 @@
===============================================
Release notes for the Genode OS Framework 15.08
===============================================
Genode Labs
The version 15.08 marks the beginning of Genode as day-to-day OS as one of the
project's core developers switched to using Genode/NOVA on his machine,
stressing the OS infrastructure we created over the course of the last seven
years. Thanks to components like VirtualBox, the Noux runtime for GNU software,
the Linux wireless stack and Rump-kernel-based file systems, the transition
went actually much smoother than expected. So other members of the team plan
to follow soon. Section [Genode as day-to-day operating system] gives an
overview of the taken approach. Genode's use as general-purpose OS provided
the incentive for most of the improvements featured by the current release,
starting with the addressing of the long-standing kernel-memory management
deficiencies of the NOVA kernel (Section [NOVA kernel-resource management]),
over enhancements of Genode's tracing and file-system facilities, to vast
improvements of the guest-host integration of VirtualBox when running on
Genode.
The release is accompanied with a second line of work led by our friends
at Codelabs: Enabling Genode to run on top of their Muen separation
kernel as described in Section [Genode on top of the Muen Separation Kernel].
Muen is a low-complexity kernel for the 64-bit x86 architecture that
statically partitions the machine into multiple domains. In contrast to
microkernels like the ones already supported by Genode, the assignment
of physical resources (such as memory, CPU time, and devices) happens at
system-integration time. Since an isolation kernel does not have to deal
with dynamic resource management at runtime, it is less complex than
a general-purpose microkernel. This makes it relatively easy to reason about
its strong isolation properties, which, in turn, makes it attractive for
high-assurance computing. With Genode being able to run within a Muen
domain, the rich component infrastructure of Genode can be combined with
the strong isolation guarantees of Muen.
Genode on top of the Muen Separation Kernel
###########################################
_This section was written by Adrian-Ken Rueegsegger and Reto Buerki who_
_conducted the described line of work independent from Genode Labs._
After completing our x86_64 port of the Genode base-hw kernel, which was
featured in the
[http://genode.org/documentation/release-notes/15.05#Principal_support_for_the_64-bit_x86_architecture - previous release (15.05)],
we immediately started working on our main goal: running a Genode system as
guest on the Muen Separation Kernel (SK). This would enable the Muen platform
to benefit from the rich ecosystem of Genode.
For those who have not read the 15.05 Genode release notes, [http://muen.sk - Muen]
is an Open-Source microkernel, which uses the [http://spark-2014.org/ - SPARK]
programming language to enable light-weight formal methods for high assurance.
The 64-bit x86 kernel, currently consisting of a little over 5'000 LOC, makes
extensive use of the latest Intel virtualization features and has been formally
proven to contain no runtime errors at the source-code level.
The new 'hw_x86_64_muen' platform, as the name implies, extends the 'hw_x86_64'
base-hw kernel by replacing the PIC and timer drivers with paravirtualized
variants.
In contrast to other kernels supported by Genode, the architecture with Muen is
different in the sense that the entire 'hw_x86_64_muen' Genode system runs as
guest VM in VMX non-root mode on the SK. From the perspective of Muen, Genode
is executed on top of the kernel like any other guest OS without special
privileges.
[image muen_system_overview]
Genode running on top of the Muen Separation Kernel alongside other subjects
This loose coupling of Muen and Genode base-hw enables the robust combination
of a static, low-complexity SK with a feature-rich and extensive OS framework.
The result is a flexible platform for the construction of component-based
high-assurance systems.
People interested in giving the 'hw_x86_64_muen' platform a spin can find a
small tutorial at _repos/base-hw/doc/x86_64_muen.txt_.
NOVA kernel-resource management
###############################
For several years, the NOVA kernel has served as Genode's primary base
platform on x86. The main reasons for this choice are: the kernel provides -
among the supported x86 kernels - the richest feature set like the support of
IOMMUs, virtualization, and SMP. It also offers a clean design and a stable
kernel interface. The available kernel-interface specification and the
readable and modern source base are a pleasure to work with. Hence, Genode
Labs is able to fully commit to the maintenance and further evolution of this
kernel.
Nevertheless, since the beginning, the vanilla kernel lacks one essential
feature to reliably host Genode as user-land, namely the proper management of
the memory used by the kernel itself (in short kernel-memory management). In
the past, we already extended the kernel to free up kernel resources when
destroying kernel objects, e.g., protection domains and page-tables, threads,
semaphores, and portals. Still, on Genode/NOVA, a component may trigger
arbitrary kernel-memory consumption during RPC by delegating memory,
capabilities, or by creating other components via Genode's core component. If
the kernel memory gets depleted, the kernel panics with an "Out of memory"
message and the entire Genode scenario stops.
In principal, the consumption of kernel memory can be deliberately provoked by
a misbehaving (greedy) component. But also during the regular day-to-day usage
of Genode, can such a situation occur when the system is used in a highly
dynamic fashion. For example, compiling and linking source code within the
noux environment constantly creates and destroys protection domains, threads,
and memory mappings. Our nightly test of compiling Genode within noux triggers
this condition every once in a while.
The main issue here is that the consumption of kernel memory is not accounted
by Genode. The kernel interface does not support such a feature. Kernels like
seL4 as well as Genode's custom base-hw kernel show how this problem can be
solved.
To improve the current situation - where the overall kernel memory is a fixed
amount - we extended NOVA in the following ways: First, the NOVA kernel
accounts any kernel memory consumption per protection domain. Second, each
process has a limited amount of kernel-memory quota it can use. Last, the
kernel detects when the quota limit of a protection domain is reached.
If the third condition occurs, the kernel stops the offending thread and
(optionally) notifies a handler thread. This so called out-of-memory (OOM)
handler thread receives information about the current situation and may
respond to it in the following ways:
* Stop the thread of the depleted protection domain, or
* Transfer kernel-memory quota between protection domains (upgrading the limit
if desired), or
* Free up kernel memory if possible, e.g., revoke memory delegations, which
can be re-created.
We implemented the steps above inside the NOVA kernel and extended Genode's
core component to handle such OOM situations. All system calls beside the IPC
call/reply may now return an error code upon depletion of the quota. Most of
these system calls can solely be performed by core and are handled inside
core's NOVA-specific platform code.
In the case of IPC call/reply operations, we desired to handle OOM cases
transparently to Genode user-level components. Therefore, each thread in
Genode/NOVA now gets constructed with an OOM IPC portal attached. This portal
is served by the pager thread in core and is traversed on OOM occurrences
during IPC operations. If a pager thread receives such an OOM IPC, it decodes
the involved IPC sender and IPC receiver and locates the appropriate
core-internal paging objects. The currently implemented out-of-memory policy
tries to upgrade the quota. If this is not possible, an attempt to revoke
memory mappings from the OOM-causing protection domain is made. This
implicitly frees-up some kernel memory (e.g., mapping nodes). If none of the
responses suffices, the handler stops the OOM-causing thread and writes a
message to the system log.
The current policy implementation constitutes a rather rough heuristic, which
may not suffice under all circumstances. In the future, we would like to
specify a distinct policy per component, e.g. depending on prior known memory
usage patterns. For example, some components follow well-known usage patterns
and therefore a fixed upper quota limit can be specified. Other components are
highly dynamic and desire quota upgrades on demand. There are many more
combinations imaginable.
Our current plan is to collect more experience over the next months with this
new kernel mechanism. Based on our observations, we may externalize such
policy decisions and possibly make them configurable per component.
The current implementation however, already avoids the situation that the
kernel goes out of service if a single component misbehaves
kernel-memory-wise.
Genode as day-to-day operating system
#####################################
At the beginning of June, Genode reached the probably most symbolic milestone
in the project's history: Norman - one of the core developers - replaced his
Linux-based working environment with a Genode-based system. This system is
composed of the following ingredients:
[image turmvilla_scenario]
The machine used is a Lenovo Thinkpad X201. We settled on this five-year-old
machine for several reasons. First, it is a very solid platform with a nice
form factor. Second, it features Intel's AMT (Active Management Technology),
which is handy to obtain low-level system logs in the case something goes
wrong. Third, refurbished machines of this type can be obtained for as little
as 200 EUR. Finally, an older machine reinforces the need for good performance
of the operating system. So it creates a natural incentive for Norman to find
and address performance bottlenecks.
Our modified version of the NOVA microhypervisor is the used kernel.
The user interface is based on our custom GUI stack including the nitpicker
GUI server as well as the window manager and its companion components
(decorator, layouter, pointer) we introduced in
[http://genode.org/documentation/release-notes/14.08#New_GUI_architecture - version 14.08].
The display is driven by the VESA driver. User input is handled by the PS/2
driver for handling the laptop keyboard and trackpoint, and the USB driver for
handling an externally connected keyboard and mouse.
Network connectivity is provided by our port of the Intel Wireless stack that
we introduced with the version
[http://genode.org/documentation/release-notes/14.11#Intel_wireless_stack - 14.11].
Our custom AHCI driver provides access to the physical hard disk. File-system
access is provided by our
[http://genode.org/documentation/release-notes/14.02#NetBSD_file_systems_using_rump_kernels - Rump-kernel-based file-system server].
A simple Genode shell called CLI monitor allows the user to start and kill
subsystems dynamically. Initially, the two most important subsystems are
VirtualBox and Noux.
VirtualBox executes a GNU/Linux-based guest OS that we refer to as "rich OS".
The rich OS serves as a migration path from GNU/Linux to Genode. It is used
for all tasks that cannot be accomplished directly on Genode yet. At the
beginning of the transition, the daily routine still very much depends on the
rich OS. By moving more and more functionality over to the Genode world, we
will eventually be able to make the rich OS obsolete step by step. Thanks to
VirtualBox' excellent host-guest-integration features, the VirtualBox window
can be dynamically resized and the guest mouse cursor integrates seamlessly
with Genode's pointer. VirtualBox is directly connected to the wireless
network driver. So common applications like Firefox can be used.
The noux runtime allows us to use command-line-based GNU software directly on
Genode. Coreutils and Bash are used for managing files. Vim is used for
editing files. Unlike the rich OS, the noux environment has access to the
Genode partition of the hard disk. In particular, it can be used to update the
Genode system. It has access to a number of pseudo files that contain status
information of the underlying components, e.g., the list of wireless access
points. Furthermore, it has limited access to the configuration interfaces of
the base components. For example, it can point the wireless driver to the
access point to use, or change the configuration of the nitpicker GUI server
at runtime.
As a bridge between the rich OS and the Genode world, we combine VirtualBox'
shared-folder mechanism with Genode's VFS infrastructure. The shared folder is
represented by a dedicated instance of a RAM file system, which is mounted in
both the VFS of VirtualBox and the VFS of noux.
As evidenced by Norman's use since June, the described system setup is
sufficient to be productive. So other members of the Genode team plan to
follow in his footsteps soon. At the same time, the continued use of the
system from day to day revealed a number of shortcomings, performance
limitations, and rough edges, which we eventually eliminated. It goes without
saying that this is an ongoing effort. Eating our own dog food forces us to
address the right issues to make the daily life more comfortable.
Feature-wise the switch to Genode motivated three developments, namely the
enhancement of Genode's CLI monitor, the improvement of the window manager,
and the creation of a CPU-load monitoring tool.
Interactive management of subsystem configurations
==================================================
The original version of CLI monitor obtained the configuration data of its
subsystems at start time via the Genode::config mechanism. But for managing
complex scenarios, the config node becomes very complex. Hence, it is
preferable to have a distinct file for each subsystem configuration.
The new version of CLI monitor scans the directory '/subsystems' for files
ending with ".subsystem". Each file has the same syntax as the formerly used
subsystem nodes. This change has the welcome implication that subsystem
configurations can be changed during the runtime of the CLI monitor, e.g., by
using a concurrently running instance of noux with access to the _subsystems/_
directory. This procedure has become an essential part of the daily work flow
as it enables the interactive evolution of the Genode system.
Window-management improvements
==============================
To make the window manager more flexible while reducing its complexity at the
same time, we removed the formerly built-in policy hosting the decorator and
layout components as children of the window manager. Those components are no
longer child components but siblings. The relationship of the components is
now solely expressed by the configuration of their common parent, i.e., init.
This change clears the way to dynamically replace those components during
runtime (e.g., switching between different decorators).
To improve the usability of the windowed GUI, we enabled the layouter to
raise windows on click and to let the keyboard focus follow the pointer.
Furthermore, the window manager, the decorator, and the floating window
layouter became able to propagate the usage of an alpha channel from the
client application to the decorator. This way, the decorator can paint the
decoration elements behind the affected windows, which would otherwise be
skipped. Consequently, partially transparent windows can be properly displayed.
CPU-load monitoring
===================
During daily system use, we started to wish to know in detail where the CPU
cycles are spent. For example, the access of a file by the rich OS involves
several components, including the guest OS itself, VirtualBox, rump_fs (file
system), part_blk (partition access), ahci_drv (SATA device access), core, and
NOVA. Investigating performance issues requires a holistic view of all those
components. For this reason, we enhanced our existing tracing infrastructure
(Section [Enhanced tracing facilities]) to allow the creation of CPU-load
monitoring tools. The first tool in this category is the graphical CPU-load
monitor located at _gems/app/cpu_load_display/_, which displays a timeline of
the CPU load where each thread is depicted with a different color. Thanks to
this tool, we have become able to explore performance issues in an interactive
way. In particular, it helped us to identify and resolve a long-standing
inaccuracy problem in our low-level timer service.
Base framework and low-level OS infrastructure
##############################################
Improved audio support
======================
In the previous release, we replaced our old audio driver with a new one that
provided the same audio-out session interface. Complementing the audio-out
session, we are now introducing a new audio-in session interface that can be
used to record audio frames. It is modeled after the audio-out interface in
the way how it handles the communication between the client and the server. It
uses shared memory in the form of the Audio_in::Stream to transport the frames
between the components. A server component captures frames and puts them into
a packet queue, which is embedded in the Audio_in::Stream. The server
allocates packets from this queue to store the recorded audio frames. If the
queue is already full, the server will override already allocated packets and
will notify the client by submitting an 'overrun' signal. The client has to
cope with this situation, e.g., by consuming packets more frequently. A client
can install a signal handler to respond to a progress signal, which is sent by
the server when a new Audio_in::Packet has been submitted to the packet queue.
For now, all audio-in server components only support one channel (left)
although the audio-in session interface principally supports multiple
channels.
The _dde_bsd_ audio_drv is the first and currently only audio driver component
that was extended to provide the audio-in session. To express this fact, the
driver was renamed from _audio_out_drv_ to _audio_drv_. In contrast to its
playback functionality, which is enabled by default, recording has to be
enabled explicitly by setting the configuration attribute 'recording' to
'yes'. If the need arises, playback may be disabled by setting 'playback' to
'no'. In addition, it is now possible to configure the driver by adjusting the
mixer in the driver's configuration node. For the time being, the interface as
employed by the original OpenBSD mixer utility is used.
The following snippet shows how to enable and configure recording on a
Thinkpad X220 where the headset instead of the internal microphone is used as
source:
! <start name="audio_drv">
! <resource name="RAM" quantum="8M"/>
! <provides>
! <service name="Audio_out"/>
! <service name="Audio_in"/>
! </provides>
! <config recording="yes">
! <mixer field="outputs.master" value="255"/>
! <mixer field="record.adc-0:1_source" value="sel2"/>
! <mixer field="record.adc-0:1" value="255"/>
! </config>
! </start>
In addition to selecting the recording source, the playback as well as the
recording volume are raised to the maximum. Information about all available
mixers and settings in general may be obtained by specifying the 'verbose'
attribute in the config node.
The enriched driver is accompanied by a simple monitor application, which
directly plays back all recorded audio frames and shows how to use the
audio-in session. It can be tested by executing the
_repos/dde_bsd/run/audio_in.run_ run script.
There are also changes to the audio-out session itself. The length of a period
was reduced from 2048 to 512 samples to accommodate for a lower latency when
mixing audio-out packets. A method for invalidating all packets in the queue
was also added.
File-system infrastructure
==========================
Unlike traditional operating systems that rely on a global name space for
files, each Genode component has a distinct view on files. Many low-level
components do not even have the notion of files. Whereas traditional operating
systems rely on a virtual file system (VFS) implemented in the OS kernel,
Genode's VFS has the form of a library that can optionally be linked to a
component. The implementation of this library originated from the noux runtime
introduced in version
[http://genode.org/documentation/release-notes/11.02#Noux_-_an_execution_environment_for_the_GNU_userland - 11.02],
and was later integrated into our C runtime in version
[http://genode.org/documentation/release-notes/14.05#Per-process_virtual_file_systems - 14.05].
With the current release, we take the VFS a step further by making it
available to components without a C runtime. Thereby, low-complexity
security-sensitive components such as CLI monitor become able to benefit from
the powerful VFS infrastructure.
The VFS itself received a welcome improvement in the form of private RAM file
systems. A need for process-local storage motivated a conversion of the
existing ram_fs server component to an embeddable VFS file system. This
addition to the set of VFS plugins enables components to use temporary file
systems without relying on the resources of an external component.
Unified networking components
=============================
Having had a good experience with our Block::Driver implementation, which
wraps the block-session interface and takes care of the packet-stream
handling, thus easing the implementation of driver and other block components,
we observed that this approach did not provide enough flexibility for
NIC-session servers. For example, NIC servers are bi-directional and when a
network packet arrives the server has to make sure that there are enough
resources available to dispatch the network packet to the client. This has to
be done because the server must never block, e.g., by waiting for allocations
to succeed or for an empty spot in the packet queue of a client. Therefore,
such a non-blocking NIC server needs to validate all preconditions for
dispatching the packet in advance and, if they cannot be met, drop the network
packet.
In order to implement this kind of behavior, NIC-session servers must have
direct access to the actual NIC session. For this reason, we removed the
Nic::Driver interface from Genode and added a Nic::Session_component that
offers common basic packet-stream-signal dispatch functionality. Servers may
now inherit from this component and implement their own policy.
We adjusted all servers that implement NIC sessions to the new interface
(dde_ipxe, wifi, usb, nic_bridge, OpenVPN, ...), and thereby unified all
networking components within Genode.
Enhanced tracing facilities
===========================
Recent Genode-based system scenarios like the one described in Section
[Genode as day-to-day operating system] consist of dozens of components that
interact with each other. For reasoning about the behaviour of such scenarios
and identifying effective optimization vectors, tools for gathering a holistic
view of the system are highly desired.
With the introduction of our light-weight
[http://genode.org/documentation/release-notes/13.08#Light-weight_event_tracing - event-tracing facility]
in version 13.08, we laid the foundation for such tools. The current release
extends core's TRACE service with the ability to obtain statistics about CPU
utilization. More specifically, it enables clients of core's TRACE service to
obtain the execution times of trace subjects (i.e., threads). The execution
time is delivered as part of the 'Subject_info' structure. In addition to the
execution time, the structure delivers the information about the affinity of
the subject with a physical CPU.
At the current stage, the feature is available solely on NOVA since this is
our kernel of choice for using Genode as our day-to-day OS. On all other base
platforms, the returned execution times are 0. To give a complete picture of
the system's threads, the kernel's idle threads (one per CPU) are featured as
trace subjects as well. Of course, idle threads cannot be traced but their
corresponding trace subjects allow TRACE clients to obtain the idle time of
each CPU.
By obtaining the trace-subject information in periodic intervals, a TRACE
client is able to gather statistics about the CPU utilization attributed to
the individual threads present (or no longer present) in the system. One
instance of such a tool is the new trace-subject reporter located at
_os/src/app/trace_subject_reporter_. It acts as a TRACE client, which delivers
the gathered trace-subject information in the form of XML-formatted data to a
report session. This information, in turn, can be consumed by a separate
component that analyses the data. In contrast to the low-complexity
trace-subject reporter, which requires access to the privileged TRACE services
of core, the (potentially complex) analysing component does not require access
to core's TRACE service. So it isn't as critical as the trace-subject monitor.
The first representative of a consumer of trace-subject reports is the
CPU-load display mentioned in Section [CPU-load monitoring] and depicted in
Figure [nano3d].
In addition to the CPU-monitoring additions, the tracing facilities received
minor refinements. Up to now, it was not possible to trace threads that use a
CPU session other than the component's initial one. A specific example is
VirtualBox, which employs several CPU sessions, one for each priority. This
problem has been solved by associating the event logger of each thread with
its actual CPU session. Consequently, the tracing mechanism has become able to
trace VirtualBox, which is pivotal for our further optimizations.
Low-complexity software rendering functions
===========================================
Our ambition to use Genode as our day-to-day OS raises the need for custom
graphical applications. Granted, it is principally possible to base such
applications on Qt5, which is readily available to native Genode components.
However, for certain applications like status displays, we prefer to avoid the
dependency on an overly complex GUI tool kit. To accommodate such
applications, Genode hosts a small collection of low-complexity graphics
functions called painters. All of Genode's low-complexity graphical components
such as nitpicker, launchpad, window decorator, or the terminal are based on
this infrastructure.
With the current release, we extend the collection with two new painters
located at _gems/include/polygon_gfx_. Both draw convex polygons with an
arbitrary number of points. The shaded-polygon painter interpolates the color
and alpha values whereas the textured-polygon painter applies a texture to the
polygon. The painters are accompanied by simplistic 3D routines located at
_gems/include/nano3d/_ and a corresponding example (_gems/run/nano3d.run_).
[image nano3d]
With the nano3d demo and our new CPU load display, the screenshot above shows
two applications that make use of the new graphics operations.
Device drivers
##############
Completing the transition to the new platform driver
====================================================
Until now, the platform driver on x86-based machines was formed by the ACPI
and PCI drivers. The ACPI driver originally executed the PCI driver as a slave
(child) service. The ACPI driver parsed the ACPI tables and provided the
relevant information as configuration during the PCI-driver startup. We
changed this close coupling to the more modern and commonly used
[http://genode.org/documentation/release-notes/14.02#New_session_interface_for_status_reporting - report_rom mechanism].
When the new ACPI driver finishes the ACPI table parsing, it provides the
information via a report to any interested and registered components. The
report contains among other the IRQ re-routing information. The PCI driver is
a component, which - according to its session routing configuration - plays
the role of a consumer of the ACPI report.
With this change of interaction of ACPI and PCI driver, the policy for devices
must be configured solely at the PCI driver and not at the ACPI driver. The
syntax, however, stayed the same as introduced with release 15.05.
Finally, the PCI driver 'pci_drv' got renamed to 'platform_drv' as already
used on most ARM platforms. All files and session interfaces containing
PCI/pci in the names were renamed to Platform/platform. The x86 platform
interfaces moved to _repos/os/include/platform/x86/_ and the implementation of
the platform driver to _repos/os/src/drivers/platform/x86/_.
An example x86 platform configuration snippet looks like this:
!<start name="acpi_drv" >
! <resource .../>
! <route>
! ...
! <service name="Report"> <child name="acpi_report_rom"/> </service>
! </route>
!</start>
!
!<start name="acpi_report_rom" >
! <binary name="report_rom"/>
! <resource .../>
! <provides> <service name="ROM" /> <service name="Report" /> </provides>
! <config>
! <rom> <policy label="platform_drv -> acpi" report="acpi_drv -> acpi"/> </rom>
! </config>
! <route> ... </route>
!</start>
!
!<start name="platform_drv" >
! <resource name="RAM" quantum="3M" constrain_phys="yes"/>
! <provides> <service name="Platform"/> </provides>
! <route>
! <service name="ROM">
! <if-arg key="label" value="acpi"/> <child name="acpi_report_rom"/>
! </service>
! ...
! </route>
! <config>
! <policy label="ps2_drv"> <device name="PS2"/> </policy>
! <policy label="nic_drv"> <pci class="ETHERNET"/> </policy>
! <policy label="fb_drv"> <pci class="VGA"/> </policy>
! <policy label="wifi_drv"> <pci class="WIFI"/> </policy>
! <policy label="usb_drv"> <pci class="USB"/> </policy>
! <policy label="ahci_drv"> <pci class="AHCI"/> </policy>
! <policy label="audio_drv"> <pci class="AUDIO"/> <pci class="HDAUDIO"/> </policy>
! </config>
!</start>
In order to unify and simplify the writing of run scripts, we added the
commonly used platform configuration to the file
_repos/base/run/platform_drv.inc_. This file may be included by any test run
script in order to setup a default platform driver configuration.
In addition, the snippet provides the following functions:
'append_platform_drv_build_components', 'append_platform_drv_config' and
'append_platform_drv_boot_modules'. The functions add necessary information to
the 'build_components', 'config' and 'boot_modules' run variables. The
_platform_drv.inc_ also contains the distinction between various ARM/x86
platforms and includes the necessary pieces. Hence, run scripts are largely
relieved from platform-specific peculiarities.
The body of an example run script looks like this:
! set build_components { ... }
!
! source ${genode_dir}/repos/base/run/platform_drv.inc
! append_platform_drv_build_components
!
! build $build_components
!
! create_boot_directory
!
! set config { ... }
!
! append_platform_drv_config
!
! append config { ... }
!
! install_config $config
!
! append_platform_drv_boot_modules
!
! build_boot_image $boot_modules
!
! run_genode_until ...
BCM57cxx network cards
======================
During Hack'n Hike 2015, we had access to a server that featured a Broadcom
network card. Therefore Guido Witmond performed the first steps to enable
Broadcom's BCM 57cxx cards. With this preliminary work in place, we were
quickly able to perform the additional steps required to add BCM 57cxx support
to Genode.
VESA driver refinements
=======================
The VESA driver now reports the frame buffer's line width instead of the
visible width to the client. This fixes a possible distortion if these widths
differ, at the cost that content in the right-most area might be invisible in
such cases.
VirtualBox
##########
Policy-based mouse pointer
==========================
In the previous release, we implemented support for the transparent
integration of the guest mouse pointer with nitpicker via the VirtualBox guest
additions and the vbox_pointer component, which is capable of rendering
guest-provided mouse-pointer shapes. Now, we extended vbox_pointer by a
policy-based configuration that allows the selection of ROMs containing the
actual mouse shape based on the nitpicker session label or domain. With this
feature in place, it is possible to integrate several VirtualBox instances as
well as dedicated pointer shapes for specific components. To see the improved
vbox_pointer in action give _run/vbox_pointer_ a shot.
Dynamic adaptation to screen size changes
=========================================
VirtualBox now notifies the guest operating system about screen-size changes
(for example if the user resizes a window, which shows the guest frame
buffer). The VirtualBox guest additions can use this information to adapt the
guest frame buffer to the new size.
SMP support
===========
Guest operating systems can now use multiple virtual CPUs, which are mapped to
multiple host CPUs. The number of virtual CPUs can be configured in the
'.vbox' file.
Preliminary audio support
=========================
At some point, the use of VirtualBox as a stop-gap solution for using Genode
as everyday OS raises the need to handle audio. With this release, we address
this matter by enabling preliminary audio support in our VirtualBox port. A
back end that uses the audio-out and audio-in sessions to playback and record
sound samples has been added. It disguises itself as the OSS back end that is
already used by vanilla VirtualBox. Since Genode pretends to be FreeBSD in the
eyes of VirtualBox (because Genode's libc is based on FreeBSD's libc), the
provisioning of an implementation of the OSS back end as used on FreeBSD host
systems is the most natural approach. The audio support is complemented by
adding the necessary device models for the virtual HDA as well as the AC97
devices to our VirtualBox port.
For now, it is vital to have the guest OS configure the virtual device in a
way that considers the current implementation. For example, we cannot
guarantee distortion-free playback or recording if the guest OS uses a period
that is too short, typically 10ms or less. There are also remaining issues
with the mixing/filtering code in VirtualBox. Therefore, we bypass it to
achieve better audio quality. As a consequence, the device model of the VM has
to use the same sample rate as is used by the audio-out and audio-in sessions
(44.1kHz).
Enabling audio support is done be adding
! <AudioAdapter controller="HDA" driver="OSS" enabled="true"/>
to the .vbox file manually or configuring the VM accordingly by using the GUI.
Platforms
#########
Execution on bare hardware (base-hw)
====================================
Bender chain loader on base-hw x86_64
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
On Intel platforms, we use the Bender chain loader from the
[https://github.com/alex-ab/morbo - Morbo multiboot suite] to detect available
COM ports of PCI plug-in cards, the AMT SOL device, or as fall back the
default comport 1. The loader stores the I/O port information of the detected
cards into the BIOS data area (BDA), from where it is retrieved by core on
boot and subsequently used for logging. With this release, we added the BDA
parsing to base-hw on x86-64 and enabled the feature in the run tool. As a
prerequisite, we had to fix an issue in bender triggered by the loading of
only one (large) multi-boot kernel. Consequently, its binary in
_tool/boot/bender_ was updated.
Revised page-table handling
~~~~~~~~~~~~~~~~~~~~~~~~~~~
One of the main advantages of the base-hw platform is that the memory trading
concept of Genode is universally applied even with regard to kernel objects.
For instance, whenever a component wants to create a thread, it pays for the
thread's stack, UTCB, and for the corresponding kernel object. The same
applies to objects needed to manage the virtual address space of a component
with the single exception of page tables.
Normally, when the quota, which was donated by a component to a specific
service, runs out, the component receives an exception the next time it tries
to invoke the service. The component can respond by upgrading the respective
session quota. However, in the context of page-fault resolution, this is
particularly difficult to do. The allocation and thereby the shortage of
memory becomes evident only when the client produces a page fault. Therefore,
there is no way to inform the component to upgrade its session quota before
resolving the fault.
Instead of designing a sophisticated protocol between core and the other
components to solve this problem, we decided to simplify the current
page-fault resolution by using a static set of page-tables per component.
Formerly, page tables were dynamically allocated from core's memory allocator.
Now, an array of page tables gets allocated during construction of a
protection domain. When a component runs out of page tables, all of its
mappings get flushed, and the page tables are populated from scratch. This
change greatly simplifies the page-table handling inside of base-hw.
Dynamic interrupt mode setting on x86_64
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
On x86-based hardware, user-level device drivers have become able to specify
the trigger mode and polarity of the interrupts when requesting an IRQ
session. On ARM, those session parameters are ignored. This change enables the
x86_64 platform to support devices, which use arbitrary trigger modes and
polarity settings, e.g., AHCI on QEMU and real hardware.
Fiasco.OC
=========
Genode's device-driver support when using the Fiasco.OC kernel as base
platform received an upgrade.
First, principle support for the Raspberry Pi was added. To make this platform
useful in practice, a working USB driver is important. I.e., the network
interface is connected via USB. Hence the USB driver got enabled for
Fiasco.OC, too. As a result, Genode's software stack can now be used on the
Raspberry Pi by using either our custom base-hw kernel or Fiasco.OC.
Second, support for the Odroid-X2 platform using the Exynos4412 SoC was added,
which includes the drivers for clock management (CMU), power management
(PMU) as well as USB.
Thanks to Reinier Millo Sánchez and Alexy Gallardo Segura for having
contributed this line of work.
Removal of deprecated features
##############################
We dropped the support for the *ARM Versatile Express* board from the Genode
source tree to relieve our automated testing infrastructure from supporting a
platform that remained unused for more than two years.
The device driver environment kit (DDE Kit) was originally intended as a
common API among the execution environments of ported user-level device
drivers. However, over the course of the past years, we found that this
approach could not fulfill its promise while introducing a number of new
problems. We reported our experiences in the release notes of versions
[http://genode.org/documentation/release-notes/12.05#Re-approaching_the_Linux_device-driver_environment - 12.05] and
[http://genode.org/documentation/release-notes/14.11#Roundup - 14.11].
To be able to remove the DDE-Kit API, we reworked the USB driver, our port of
the Linux TCP/IP stack, and the wireless driver accordingly.

1429
doc/release_notes-15-11.txt
View File

File diff suppressed because it is too large Load Diff

652
doc/release_notes-16-02.txt
View File

@@ -1,652 +0,0 @@
===============================================
Release notes for the Genode OS Framework 16.02
===============================================
Genode Labs
With version 16.02, we add RISC-V to Genode's supported CPU architectures,
enable the secure pass-through of individual USB devices to virtual machines,
and update the support for the Muen and seL4 kernels.
Trustworthy hardware becomes an increasingly pressing problem. With each new
generation of today's commodity hardware comes a dramatic increase of
complexity, the addition of proprietary companion processors, and opaque
firmware blobs. Even with a perfectly secure operating system, the user's
privacy and security remains at risk as there is no way to assess the
trustworthiness of our underlying hardware. RISC-V is a new hardware
architecture that tries to overcome this problem by the means of open source
and transparency. It is designed to scale from micro controllers to
general-purpose computers, and to be both synthesizable as FPGA softcores and
implementable in ASICs. The prospect of a scalable and trustworthy open-source
hardware platform motivated us to add RISC-V to Genode's supported CPU
architectures. Section [New support for the RISC-V CPU architecture] gives a
brief overview of this line of work.
Thanks to the growing number of our regular developers using Genode as day to
day OS, we create a natural incentive to address typical desktop-OS work
flows. In particular, the new version comes with the ability to assign
individual USB devices to VirtualBox instances. Conceptually, this looks like
a relatively straight-forward feature. But as discussed in Section
[Assignment of USB devices to virtual machines], we had to overcome a number of
challenging problems caused by the inherently dynamic nature of USB-device
hot-plugging. Also on the account of day-to-day computing, the GUI stack
received welcomed usability improvements like keyboard shortcuts for certain
window-management operations.
With respect to Genode's underlying base platforms, we are happy to announce
the updates of the Muen and seL4 kernels. The Muen separation kernel received
an update to version 0.7, which accommodates Genode's regular work flows (via
run scripts) much better than the previous version. As described in Section
[Muen separation kernel], this change clears the way to subject Muen to
Genode's regular automated tests. The seL4 kernel represents an exciting
playground as a future base platform for Genode. We have updated the kernel to
version 2.1, which prompted us to fundamentally revisit the low-level resource
management of Genode on this kernel. A summary of this undertaking is presented
in Section [seL4 version 2.1].
According to the [http:/about/road-map - road map], we originally planned to
revise the framework API in this release. Even though this topic is
[https://github.com/genodelabs/genode/issues/1832 - very actively pursued], we
decided to not rush it. We find it important to provide a smooth migration path
from the old API to the new one. Determining the best path is actually trickier
than revising the API, though. To let our decisions settle a bit, we postpone
the transition to the upcoming release.
Assignment of USB devices to virtual machines
#############################################
As a migration strategy for running Genode on a daily basis, using VirtualBox
to execute a feature-rich OS is vital. In release
[http://genode.org/documentation/release-notes/15.05#USB-device_pass-through_support - 15.05],
we added USB pass-through support to VirtualBox by enabling its integrated USB
proxy service. Since we use the open-source edition of VirtualBox, we were
merely able to use the OHCI device model and were therefore limited to using
USB 1.x devices in low and full speed mode only. To make matters worse, when
using the OHCI controller model, it is difficult if not impossible to access
USB mass-storage devices. Usually, VirtualBox facilitates the EHCI or xHCI
device models for the pass-through of storage devices. Unfortunately, those
models are only available as a proprietary extension, which cannot be used by
our VirtualBox port.
Having support for the pass-through of high-speed and super-speed USB devices
is a must in such controller models. Therefore, we either have to implement
these models ourselves or port existing ones from another VMM or emulator to
fill the gap. We went for porting existing models first because device-model
development from scratch could end up being time consuming if we want to
guarantee them to work with a variety of different OS drivers.
QEMU xHCI device model
----------------------
QEMU features a NEC xHCI (UPD720200) device model that works well with Windows
guests. For this reason, we decided to give porting this device model a shot.
We applied the DDE approach and started by creating a QEMU emulation
environment so that only the bare minimum amount of source code needed to be
taken from the QEMU sources. It came down to a handful of source files, mainly
the USB core and the xHCI device model files. We iteratively extended the
emulation environment until the QEMU sources compiled and linked fine. One
particular cumbersome issue we had to overcome was the emulation of the QEMU
Object Model. Since QEMU is written in C, it uses its own object model to
implement inheritance. This object model is used throughout QEMU. We took the
easy way out and just used a C++ wrapper class that contains all QEMU objects
that are used in the USB subsystem.
The next step was to develop a USB host device model. This model connects a
USB device attached to Genode's USB host-controller driver to the xHCI device
model. Lucky for us, QEMU already contains a USB host device model that uses
libusb, which we could use as blueprint. We implemented a USB host device that
leverages Genode's custom USB session interface. This host device reacts to a
USB device report coming from another component such as the host-controller
driver. It tries to claim all devices it finds in that report and then creates
a QEMU USB device for each of them that is attached to the xHCI device model.
The xHCI device model needs infrastructure that normally is provided by QEMU
itself such as a timer queue and PCI device handling. We introduced a QEMU
USB controller interface _repos/libports/include/qemu/usb.h_ whose back-end
library interface has to be implemented by a component, i.e. the VMM, that
wants to use the library.
In the end, this work resulted in a small library that contains the xHCI
device model and works in a standalone way. All required resources have to be
provided by the component using the library. This makes it easy to integrate
the library in different VMMs because the user of the library is not forced to
employ the library in a certain way but free to use it any way he chooses.
xHCI device model wrapper in VirtualBox
---------------------------------------
We implemented an xHCI device model _repos/port/src/virtualbox/devxhci.cc_ in
VirtualBox that merely wraps the QEMU USB library and provides the back-end
functionality required by the library to glue QEMU's xHCI device model to
VirtualBox. For now, this device is always part of a VM because there is
currently no way to disable it from within the VirtualBox configuration
front end. Therefore, it is necessary to always give VirtualBox access to a
_usb_devices_ ROM module.
We removed the afore mentioned USB proxy service from our VirtualBox port
because it became redundant with the advent of our xHCI device model.
USB device report filter
------------------------
With the xHCI support in VirtualBox in place, we had to come up with a
mechanism to select, which USB devices it may access. Since USB devices are
usually hot-plugged by the user of the system, we need to be able to configure
the access permissions dynamically at run-time. On this account, we created a
component that intercepts the report from the USB host-controller driver. On
the one hand, this USB device report-filter component screens the device
report coming from the USB host-controller driver by checking each reported
device against a given white list of devices. Only approved devices are
reported to a consumer of the report, i.e. VirtualBox. On the other hand, this
component generates a new configuration for the USB host-controller driver.
The configuration has to be changed each time the filter component finds a
suitable device because the driver will hand out access to a given device to a
client only if there is a valid policy. As we do not know in advance, which
devices might be plugged in, this policy must be maintained dynamically. The
report filter will send the device report only if the host-controller driver
has changed its configuration. This ensures that a matching policy will be in
effect at the time when the client component tries to access the device.
The configuration of the report-filter component can also be changed at run
time.
See _repos/os/src/app/usb_report_filter/README_ for more details on how the
USB device report filter may be configured.
Example configuration
---------------------
The following figure illustrates the interplay and configuration of the
involved components:
[image qemu_xhci]
When the user plugs in a USB device, the USB host-controller driver generates
a device report that is consumed by the USB device report-filter component
(1). The filter component then examines the report and checks if it contains a
device it should report to its report consumer. It then reconfigures the
host-controller driver (2). Afterwards it sends a report to its consumer (3).
The consumer, in this case a VMM, then accesses the USB device (4).
New support for the RISC-V CPU architecture
###########################################
We became aware of [http://riscv.org - RISC-V] when attending several talks
about the project at [https://fosdem.org - FOSDEM] in 2015. RISC-V aims to be
an open-source hardware architecture and is now complemented by many projects
that target the release of real hardware or ASICs (for example,
[http://lowrisc.org - the LowRISC project]). We have experience with various
major CPU architectures and many systems on a chip and, therefore, embrace a
sharp eye on certain platform properties. Intel's ME and ARM's Trustzone
practically lock out operating systems of certain hardware and firmware
features. The true nature of these mechanisms becomes increasingly dubious,
especially when trying to build a secure open-source operating system. Intel's
AMT technology for instance comes with a complete TCP/IP stack that intercepts
packets from the integrated NIC and a VNC server that can magically expose a
mouse and a keyboard at the USB controller. If you are interested in more
details about this topic
[http://blog.invisiblethings.org/papers/2015/x86_harmful.pdf - Intel x86 considered harmful]
by Joanna Rutkowska is a very good read. We decided to have a deeper look at
the RISC-V architecture as an alternative open hardware platform. Especially,
since the LowRISC project promises a completely open system on chip, including
the peripherals.
RISC-V comes with a lot of optional features, so it can cover a large field of
applications reaching from simple I/O processors to general-purpose computing.
For example, there are 64 and 32 bit ISA (instruction set architecture)
versions, three page table formats with the option to omit paging at all, up
to four privilege modes, and a minimal integer core ISA (I). Everything else,
like multiplication and division (M), atomic instructions (A), and floating
point support (F) are subject to ISA extensions and are completely optional
for a specific hardware implementation.
For Genode, we chose to add the RISC-V support to our custom _base-hw_ kernel.
Since Genode may be used as a general purpose OS, we implemented the kernel
using the 64 bit RISC-V version, the Sv39 three-level page table format, and
the so-called general-purpose extension (G), which is the abbreviation for the
IAMF extensions. The current implementation provides the kernel and the
necessary adaptations of the user level part of core.
For testing, we used the RISC-V instruction emulator called
[https://github.com/riscv/riscv-isa-sim - Spike]. There also exists a RISC-V
implementation for various Zynq FPGAs. Genode's Zynq board support has kindly
been added and contributed by Mark Vels.
In the current state, basic Genode applications including core, init, and
components that use shared libraries can be executed on top of our RISC-V
port. We did not enable the libc and postponed further activity as the
platform currently does not specify the interaction with peripherals.
Steps to test Genode on RISC-V
------------------------------
# Building the instruction emulator
! # download the front end server
! git clone https://github.com/ssumpf/riscv-fesvr.git
!
! # build the front end server
! cd riscv-fesvr
! mkdir build
! cd build
! export RISCV=<installation path>
! ../configure --prefix=$RISCV
! (sudo) make install
!
! # download the instruction emulator
! cd ../../
! git clone https://github.com/ssumpf/riscv-isa-sim.git
! cd riscv-isa-sim
!
! # build the emulator
! mkdir build
! cd build
! ../configure --prefix=$RISCV --with-fesvr=$RISCV
! (sudo) make install
!
! # add $RISCV/bin to path
! export PATH=$RISCV/bin:$PATH
# Building Genode and running a test scenario
! # download Genode
! cd ../../
! git clone https://github.com/genodelabs/genode.git
!
! # build the Genode tool chain
! cd genode
! ./tool/tool_chain riscv
!
! # create RISC-V build directory
! ./tool/create_builddir hw_riscv
! cd build/hw_riscv
!
! # build and execute the printf run script
! make run/printf
GUI stack usability improvements
################################
Motivated by the daily use of Genode as desktop OS by an increasingly number
of developers, the window-layouter component of the
[http://genode.org/documentation/release-notes/15.11#GUI_stack - GUI stack]
received welcomed usability improvements.
Configurable window placement
-----------------------------
The policy of the window layouter can be adjusted via its configuration. For
a given window label, the window's initial position and its maximized state
can be defined as follows:
! <config>
! <policy label="mupdf" maximized="yes"/>
! <policy label="nit_fb" xpos="50" ypos="50"/>
! </config>
Keyboard shortcuts
------------------
The window layouter has become able to respond to key sequences. However,
normally, the layouter is not a regular nitpicker client but receives only
those input events that refer to the window decorations. It never owns the
keyboard focus. In order to propagate global key sequences to the layouter,
nitpicker must be explicitly configured to direct key sequences initiated with
certain keys to the decorator. For example, the following nitpicker
configuration routes key sequences starting with the left windows key to the
decorator. The window manager, in turn, forwards those events to the layouter.
! <start name="nitpicker">
! ...
! <config>
! ...
! <global-key name="KEY_LEFTMETA" label="wm -> decorator" />
! ...
! </config>
! ...
! </start>
The response of the window layouter to key sequences can be expressed in the
layouter configuration as follows:
! <config>
! <press key="KEY_LEFTMETA">
! <press key="KEY_TAB" action="next_window">
! <release key="KEY_TAB">
! <release key="KEY_LEFTMETA" action="raise_window"/>
! </release>
! </press>
! <press key="KEY_LEFTSHIFT">
! <press key="KEY_TAB" action="prev_window">
! <release key="KEY_TAB">
! <release key="KEY_LEFTMETA" action="raise_window"/>
! </release>
! </press>
! </press>
! <press key="KEY_ENTER" action="toggle_fullscreen"/>
! </press>
! </config>
Each '<press>' node defines the policy when the specified 'key' is pressed.
It can be equipped with an 'action' attribute that triggers a window action.
The supported window actions are:
:next_window: Focus the next window in the focus history.
:prev_window: Focus the previous window in the focus history.
:raise_window: Bring the focused window to the front.
:toggle_fullscreen: Maximize/unmaximize the focused window.
By nesting '<press>' nodes, actions can be tied to key sequences. In the
example above, the 'next_window' action is executed only if TAB is pressed
while the left windows-key is kept pressed. Furthermore, key sequences can
contain specific release events. In the example above, the release of the left
windows key brings the focused window to front, but only if TAB was pressed
before.
Device drivers
##############
USB host-controller driver enhancements
=======================================
The _usb_drv_ component now solely uses a policy to grant other components
access to USB devices exposed by its raw interface (USB session). On the basis
of the 'label' attribute, it will choose a pre-configured device that is
identified by either the 'bus' and 'dev' or the 'vendor' and 'product'
attribute tuple. To accommodate policy decisions made at run time, the USB
driver is now able to reload its configuration on demand. The USB device
report now contains a 'bus' and a 'dev' attribute as well in order to identify
a USB device more precisely. In addition to that, there is also a generated
'label' attribute in form of 'usb-<bus>-<dev>' that may be used to form
policies while configuring the system dynamically, e.g., when using the
_usb_report_filter_ component.
USB mass-storage driver
=======================
Up to now, access to USB storage devices was provided by the USB
host-controller driver only. However, its ability to do so is limited. E.g.,
it only supports one storage device and the storage device cannot be changed
at run-time. With this release we add a USB mass-storage driver that supports
UMS bulk-only devices that use the SCSI Block Commands set (direct-access).
This is still most common for USB sticks. Devices using different command
sets, e.g SD/HC devices or some external disc drives, will not work properly
if at all. The driver uses the USB session interface to access the USB device
and provides its service as block session to its client.
This component is part of the first step providing the ability to mount and
use USB sticks dynamically when using Genode as a general purpose OS. In the
future, the _usb_drv_ component should solely be the host-controller driver
while other tasks are handled by dedicated USB driver components such as this
one.
Audio output on Linux
=====================
The audio-out driver for Linux was modernized by replacing its multi-threaded
architecture by an event-driven architecture using Genode's server API. In
addition, the playback is now driven by a timer. For now it is a periodic
timer that triggers every 11 ms which is roughly the current audio-out period.
The driver now also behaves like the other BSD-based audio-out driver, i.e.,
it always advances the play pointer. That is vital for the audio-out stack
above the driver to work properly (e.g., the mixer).
Libraries and applications
##########################
New Genode-world repository
===========================
With a growing number of users and contributors comes the desire to bring more
and more existing software to Genode. Most of such libraries and applications,
however, are outside of the scope of Genode as an OS framework. In contrast to
device drivers, protocol stacks, and low-level OS services, which we subject
to our regular automated tests, most 3rd-party software is pretty independent
from Genode. The attempt to integrate the growing pool of such diverse
software into the main repository does not scale.
For this reason, we introduce the new
[https://github.com/genodelabs/genode-world - Genode World] repository, which
is the designated place for hosting ported applications, libraries, and games.
To use it, you first need to obtain a clone of Genode:
! git clone https://github.com/genodelabs/genode.git genode
Now, clone the _genode-world.git_ repository to _genode/repos/world:_
! git clone https://github.com/genodelabs/genode-world.git genode/repos/world
By placing the _world_ repository under the _repos/_ directory, Genode's tools
will automatically incorporate the ports provided by the _world_ repository.
For building software of the _world_ repository, the build-directory
configuration _etc/build.conf_ must be extended with the following line:
! REPOSITORIES += $(GENODE_DIR)/repos/world
*Word of caution*
In contrast to the components found in the mainline Genode repository, the
components within the _world_ repository are not subjected to the regular
quality-assurance measures of Genode Labs. Hence, problems are to be expected.
If you encounter bugs, build problems, or stability issues, please report them
to the [https://github.com/genodelabs/genode-world/issues - issue tracker] or
the [http://genode.org/community/mailing-lists - mailing list].
Updated 3rd-party software
==========================
The following 3rd-party code packages of the _ports_ and _libports_
repositories have been ported or updated:
* Lynx 2.8.8rel.2 (noux package)
* OpenSSH 7.1p1 (noux package)
* tar-1.27 (noux package)
* libssh 0.7.2
* Lighttpd 1.4.38
Platforms
#########
Execution on bare hardware (base-hw)
====================================
Within the last months, the initialization code of our custom kernel got
re-arranged to simplify the addition of new architectures, e.g., the RISC-V
port (Section [New support for the RISC-V CPU architecture]) while also making
its implementation leaner. A positive side effect of this work was the
generalization of multi-processor and L2-cache support for ARM's Cortex-A9
CPUs. For instance, the Wandboard (Freescale i.MX6 SoC) is now driven with all
four cores, and its memory can be accessed with full speed.
Besides those feature additions, we fixed an extremely rare and tricky race
condition in the implementation of the kernel-protected capabilities,
introduced in release 15.05. A capability's lifetime within a component is
tracked by a reference-counting like mechanism that is under control of the
component itself. When the kernel transfered a capability to a component, and
the very same capability was deleted within the component simultaneously, the
received capability was marked as invalid, which led to diverse, sporadic
faults. This deficit in the capabilities reference-counting is solved with the
current release.
Muen separation kernel
======================
Build integration
-----------------
Building Genode scenarios running on top of the
[http://muen.sk - Muen separation kernel] has been greatly simplified by
properly integrating the Muen system build process into the Genode build system.
As described in the
[http://genode.org/documentation/release-notes/15.08#Genode_on_top_of_the_Muen_Separation_Kernel - 15.08 release notes],
the architecture with Muen is different since the entire hw_x86_64_muen Genode
system runs as a guest VM on top of the separation kernel. This means that the
Genode base-hw image must itself be packaged into the final Muen system image
as an additional step after the Genode system build.
The packaging process of a Muen system image is performed by the new
_image/muen_ run-tool plugin, which processes the following RUN_OPT parameters.
:--image-muen-external-build:
Muen system is built automatically or externally
:--image-muen-system:
Muen system policy
:--image-muen-components:
Muen system components required for the given system policy
:--image-muen-hardware:
Muen target hardware platform
:--image-muen-gnat-path:
Path to GNAT toolchain
:--image-muen-spark-path:
Path to SPARK toolchain
The options are automatically added to the _etc/build.conf_ file for the
hw_x86_64_muen base-hw platform. The
[http://genode.org/documentation/platforms/muen - documentation] has been
updated to reflect the new, simplified build process.
A port file was added to facilitate the download of the Muen sources v0.7 and
to check the required dependencies.
Using the new _image/muen_ script in combination with iPXE allows to run the
Genode test suite via the autopilot tool.
MSI support
-----------
Muen employs Intel VT-d interrupt remapping (IR) besides DMA remapping for
secure device assignment. As a consequence, PCI devices using Message Signaled
Interrupts (MSI) must be programmed to trigger requests in remappable format
(see Intel VT-d specification, Section 5.1.2.2 for further details).
To enable the use of MSIs with the base-hw kernel, a platform-specific
function has been introduced that returns the necessary MSI parameters for a
given PCI device. If either the platform or the specific device does not
support MSI, the function returns false.
On hw_x86_64_muen, the function consults the Muen subject info page to supply
the appropriate information to the IRQ session. This allows Genode device
drivers to transparently use MSIs for passed-through PCI devices.
seL4 version 2.1
================
By the end of 2015, the [http://sel4.systems/ - seL4 kernel] version 2.0 was
published. With the current release, we update Genode's preliminary support
for this kernel from the experimental branch of one year ago to the master
branch of version 2.1. Note that this line of work is still considered as an
exploration. As of now, there is still a way to go until we can leverage seL4
as a fully featured base platform. Under the hood of Genode, the transition to
the version 2.1 master branch had the following implications.
In contrast to the experimental branch, the seL4 master branch has no way to
manually define the allocation of kernel objects within untyped memory ranges.
Instead, the kernel maintains a built-in allocation policy. This policy rules
out the deallocation of once-used parts of untyped memory. The only way to
reuse memory is to revoke the entire untyped memory range. Consequently, we
cannot share a large untyped memory range for kernel objects of different
protection domains. In order to reuse memory at a reasonably fine granularity,
we need to split the initial untyped memory ranges into small chunks that can
be individually revoked. Those chunks are called "untyped pages". An untyped
page is a 4 KiB untyped memory region.
The bootstrapping of core has to employ a two-stage allocation approach now.
For creating the initial kernel objects for core, which remain static during
the entire lifetime of the system, kernel objects are created directly out of
the initial untyped memory regions as reported by the kernel. The so-called
"initial untyped pool" keeps track of the consumption of those untyped memory
ranges by mimicking the kernel's internal allocation policy. Kernel objects
created this way can be of any size. For example the CNode, which is used to
store page-frame capabilities is 16 MiB in size. Also, core's CSpace uses a
relatively large CNode.
After the initial setup phase, all remaining untyped memory is turned into
untyped pages. From this point on, newly created kernel objects cannot exceed
4 KiB in size because one kernel object cannot span multiple untyped memory
regions. The capability selectors for untyped pages are organized similarly to
those of page-frame capabilities. There is a new 2nd-level CNode
(UNTYPED_CORE_CNODE) that is dimensioned according to the maximum amount of
physical memory (1M entries, each entry representing 4 KiB). The CNode is
organized such that an index into the CNode directly corresponds to the
physical frame number of the underlying memory. This way, we can easily
determine an untyped page selector for any physical addresses, i.e., for
revoking the kernel objects allocated at a specific physical page. The
downside is the need for another 16 MiB chunk of meta data. Also, we need to
keep in mind that this approach won't scale to 64-bit systems. We will
eventually need to replace the PHYS_CORE_CNODE and UNTYPED_CORE_CNODE by CNode
hierarchies to model a sparsely populated CNode. The following figure
illustrates the layout of core's capability space.
[image sel4_core_cspace_master]
Organization of core's capability space on seL4
For each protection domain, core maintains a so-called VM CSpace that holds
capability selectors for page frames and page tables. The size constraint of
kernel objects has the immediate implication that the VM CSpaces of protection
domains must be organized via several levels of CNodes. I.e., as the top-level
CNode of core has a size of 2^12, the remaining 20 PD-specific CSpace address
bits are organized as a 2nd-level 2^4 padding CNode, a 3rd-level 2^8 CNode,
and several 4th-level 2^8 leaf CNodes. The latter contain the actual selectors
for the page tables and page-table entries of the respective PD.
As another slight difference from the experimental branch, the master branch
requires the explicit assignment of page directories to an ASID pool.
Functionality-wise the update to version 2.1 brings no changes. The
preliminary support is still limited to Genode's most fundamental mechanisms
like the bootstrapping, the creation of protection domains, the execution of
threads, and inter-component communication. User-level device drivers are not
supported yet. Such functional improvements are scheduled for Genode 16.08.
Linux
=====
We started to experience crashes of our dynamic linker (ldso) when using
Genode's _base-linux_ platform on recent Linux kernels. Ldso is primarily a
shared object, which is linked to dynamic binaries. But ldso is also an
executable, which, once started loads the dynamically-linked binary along with
all shared libraries required by the binary. Up to now, ldso had to be loaded
at a link address defined at compilation time, which we enforced through
linker-script magic. Unfortunately, this does not work any longer on recent
Linux versions. The kernel notices that ldso is a shared object and loads it
at an arbitrary (randomized) address, which ultimately results in a
segmentation fault during ldso initialization. We found a fix for this issue
by marking ldso as an executable in the ELF header. But since ldso is linked
to all dynamic binaries (it contains Genode's base libraries) the GNU linker
then refused to link because ldso was not marked as a shared object.
Therefore, we decided to implement true self relocation within ldso. This
feature only works on Genode's base-linux platform as it requires some
symbol-address magic.

1315
doc/release_notes-16-05.txt
View File

File diff suppressed because it is too large Load Diff

1126
doc/release_notes-16-08.txt
View File

File diff suppressed because it is too large Load Diff

729
doc/release_notes-16-11.txt
View File

@@ -1,729 +0,0 @@
===============================================
Release notes for the Genode OS Framework 16.11
===============================================
Genode Labs
In contrast to most parts of the framework, the fundamental low-level
protocols, which define the interaction between parent and child components
have remained unchanged since the very first Genode version. From this
interplay, the entire architecture follows. That said, certain initial design
choices were not perfect. They partially resulted from limitations of the
kernels we used during Genode's early years and from our pre-occupation with a
certain style of programming. Over the years, the drawbacks inherent in our
original design became more and more clear and we drafted rough plans to
overcome them. However, reworking the fundamental protocols of a system that
already accommodates hundreds of component implementations cannot be taken
lightly. Because of this discomfort, we repeatedly deferred the topic -
until now. With the rapidly growing workloads carried by Genode, we
deliberately decided to address long-standing deficiencies rather than adding
the features we originally planned according to the
[https://genode.org/about/road-map - road map].
Section [Asynchronous parent-child interactions] presents the reworking of
Genode's component interplay at the lowest level. With this change in place,
we feel much more comfortable to scale up our workloads in the upcoming
releases.
Functionality-wise, the most prominent topic of the current release is the
vastly improved NIC-routing component. Since we introduced the first version
of the NIC router in the previous release, we took an iterative approach to
shape the component according to its most prominent use cases. Section
[Further improved virtual networking] summarizes the changes and the
motivation behind them.
Even though we added support for seL4 in the previous release, the NOVA
hypervisor is still our go-to kernel for x86-based hardware because of its
feature set. For this reason, we continuously improve this kernel and the
NOVA-specific components like VirtualBox. Section [NOVA hypervisor] covers
the introduction of an asynchronous map operation to NOVA.
Further topics of the current release range from added smart-card support,
over a new timeout API, to a VFS-based time-based password generator. With
respect to the road map, we postponed most topics originally planned. In
particular, we intended to enable the use of Genode on top of Xen by following
the footsteps of the existing Muen support - using our custom
base-hw kernel within a Xen DomU domain. However, before proceeding this
route, we decided to modernize the kernel design, in particular with respect
to bootstrapping and address-space management. Some parts of this line of work
are already present in the current release, for example the unification of the
boot-module handling as explained in Section
[Unified handling of boot modules].
Asynchronous parent-child interactions
######################################
When Genode was born in 2006, the L4 microkernels of the time universally
lacked an asynchronous inter-process-communication (IPC) mechanism.
Consequently, we designed the first version of Genode with the presumption
that components had to interact solely synchronously. To us, this seemed to be
the "right" way because the synchronous low-footprint IPC was presumably the
key for L4's good performance. It felt natural to leverage this benefit to the
maximum extent possible.
To illustrate the implications of this line of thinking for Genode, let's take
a look at a simple scenario where a parent component hosts two children and one
child provides a service to the other child.
[image simple_scenario]
During the creation of a session, the kernel's IPC mechanism serves three
purposes. First, it is used to communicate information between different
protection domains, in this case the parent, the client, and the server.
Second, it implicitly dictates the flow of control between the involved
parties because the caller blocks until the callee replies to the IPC call.
Third, the IPC is the mechanism to delegate authority (like the authority to
access the server's session object) between protection domains. The latter is
realized with the kernel's ability to carry capabilities as IPC message
payload. If this sounds a bit too abstract, please consider reviewing Section
3.1. "Capability-based security" of the
[https://genode.org/documentation/genode-foundations-16-05.pdf - Genode Foundations].
Using solely a synchronous IPC mechanism, the sequence of establishing a
session in the given scenario is as follows. In the context of Genode,
we usually refer to synchronous IPC as RPC (remote procedure call).
[image sync_session_seq]
The sequence looks straightforward:
# The client issues an RPC call to its parent, requesting a session for a
service of the given type while also passing a number of session-construction
arguments along with the request.
# Given the service name as provided with the session request, the parent
determines the server to ask for a new session. It requests a session
on behalf of the client by performing an RPC call to the server's prior
registered "root" capability. This capability refers to an interface for
creating and closing sessions.
# The server responds to the invocation of its root interface by creating
a new session object along with a session capability.
Whereas the session object is local to the server, the corresponding
session capability can be passed (delegated) to other components.
Each component in possession of the session capability is able to interact
with the server's corresponding session object via RPC calls.
The server returns the session capability to the parent as the result of the
parent's RPC call.
# The parent forwards the session capability to the client as the result of
the client's original RPC call.
Even though the simplicity of this protocol seems nice, it has inherent
limitations:
First, as the parent performs a synchronous RPC call to the server on behalf
of the client, it must trust the server to eventually respond to the RPC call.
If the server doesn't, the parent may block forever. In contrast to the client
that actually uses the service and thereby relies on the liveliness of the
server, the parent should not need to trust the server to be responsive. To
deal with the risk of an unresponsive server, Genode's existing runtime
environments (like the init component), maintain a dedicated thread for each
child. The session requests originating from a child are handled by the
corresponding parent-local child thread. In the worst case - if the server
fails to respond - only a single child thread stays blocked but the other
parts of the runtime environment remain unaffected. Consequently, runtime
environments have to be multi-threaded components. This, in turn, comes at the
cost of added complexity, in particular the need for error-prone inter-thread
synchronization.
Second, the approach keeps the parent's state implicitly stored in the stacks
of the parent's threads. This becomes a problem in dynamic runtime
environments that need to kill subsystems at arbitrary times. E.g., imagine
the situation where the client component is to be destroyed while the parent's
call to the server's root interface is still pending. The safe destruction of
the child - including its associated parent-local child thread - requires the
parent to abort the RPC call, which is a complex and - again - error-prone
operation.
Third, even though not inherent to synchronous RPC, Genode's original design
facilitated the use of a session capability as argument for requesting the
parent to close a specific session. However, the use of capabilities as
re-identifiable tokens is not well supported by most kernels, including seL4
([http://sel4.systems/pipermail/devel/2014-November/000114.html - discussion]
on the seL4 mailing list).
Asynchronous communication throughout Genode
--------------------------------------------
In 2008, we acknowledged the sole reliance on synchronous RPC as too limiting
and introduced an
[https://genode.org/documentation/release-notes/8.11#Asynchronous_notifications - API for asynchronous notifications].
On the traditional L4 kernels, we implemented the API by using Genode's
core component as a proxy for signal delivery. The use of asynchronous
notifications soon became natural and wide-spread throughout Genode. Today,
most session interfaces combine three forms of inter-component communication,
namely synchronous RPC calls, asynchronous notifications, and shared memory.
The new Genode API introduced in
[https://genode.org/documentation/release-notes/16.05#The_great_API_renovation - version 16.05]
further cultivated the modeling of Genode components as single-threaded state
machines instead of multi-threaded programs.
Still, until now, the most fundamental mechanism of Genode - the protocol
between parent and child components - has remained synchronous. The reasons
are twofold. First, our workaround for realizing runtime environments in a
multi-threaded way worked too well. So we were not constantly bothered by this
design problem. Second and more importantly, redesigning the fundamental
mechanism of the framework while not breaking the more than 300 existing
components is quite scary. But in anticipation of the rapidly scaling
workloads imposed on Genode, we had to take on the problem sooner or later.
We figured that now - with the modernized framework API in place - it's the
right time. From redesigning the interplay of parent and child components, we
will become able to create single-threaded runtime environments that behave
completely deterministically while consuming less resources than
multi-threaded programs. By the explicit enumeration of possible states, we
greatly ease the validation/evaluation of such crucial components.
New session-creation procedure
------------------------------
Following the asynchronous approach, the sequence of creating a session now
looks as follows:
[image async_session_seq]
The dotted lines are asynchronous notifications, which have fire-and-forget
semantics. A component that triggers a signal does not block.
The following points are worth noting:
* Sessions are identified via IDs, which are plain numbers as opposed to
capabilities. The IDs as seen by the client and server belong to different
ID name spaces.
IDs of sessions requested by the client are allocated by the client. IDs
of sessions requested at the server are allocated by the parent.
* The parent does no longer need to perform RPC calls to any of its children.
Hence, the need for multiple threads in runtime environments disappears.
* Each activation of the parent merely applies a state change of the session's
meta data structures maintained at the parent, which capture the entire
state of session requests. There is no hidden state stored on the parent's
stack.
* The information about pending session requests is communicated from the
parent to the server via a ROM session. At startup, the server requests
a ROM session for the ROM module "session_requests" from its parent. The
parent implements this ROM session locally. Since ROM sessions support
versions, the parent can post version updates of the "session_requests"
ROM with the regular mechanisms already present in Genode.
* The involved parties can potentially run in parallel.
Outcome and current state
-------------------------
Intuitively, the sequence of steps required to establish a session has
become more complicated. However, for the users of the framework, the entire
procedure is completely transparent. With a few tricks, we were actually able
to implement this fundamental change while keeping almost all existing
components untouched. One trick is the introduction of a server-local proxy
mechanism, which translates the requests obtained from the "session_requests"
ROM to component-local RPC calls on the server's root interface. So from the
perspective of an existing server component, a session request still looks
like a synchronous RPC request from the outside. Of course, the proxy is meant
as an intermediate solution until we have crafted a convenient front-end API
for the asynchronous mode of operation.
Even though the biggest share of components remains unaffected by the change,
this is not true for all components. In particular, runtime environments had
to be reworked, in some cases quite fundamentally. These include core, init,
noux, the loader, GDB monitor, launcher, CLI monitor, and the platform driver.
The change does not only affect the interplay between components but also
required a reconsideration of the child-creation procedure.
Besides the architectural improvement, this line of work had two welcome
effects.
First, in contrast to the original design, which relied on capabilities as
re-identifiable tokens, the new version greatly alleviates the need for
re-identifying capabilities on seL4. So we are able to eliminate a
long-standing problem with Genode on this kernel.
Second, the work called for new data structures for the safe interaction with
ID spaces (_base/id_space.h_) and object registries (_base/registry.h_). Those
data structures will possibly be useful in a lot of places that currently use
plain (and fairly unsafe) AVL trees or lists.
At the API level, the change is almost transparent to regular components,
except for two details. The upgrading of session quota is no longer
possible by a mere RPC call to the parent. Instead, 'Connection' objects
received a new 'upgrade_ram' method that must be used instead. Speaking
of 'Connection' objects, we had to remove the (fairly obscure) 'KEEP_OPEN'
feature, which is conceptually incompatible with the new design.
Further improved virtual networking
###################################
The
[https://genode.org/documentation/release-notes/16.08#Virtual_networking_and_support_for_TOR - previous release]
introduced the NIC router - a component that individually routes IP
packets between multiple NIC sessions, translates between different IP
subnets, and also supports port forwarding and NAT. For the first version of
the NIC router, we focused on the technical realization. Now, besides
some optimization and restructuring, we took the chance to polish the
configuration interface of the component. The goal was to make the interface
more intuitive and reduce pitfalls to a minimum. Roughly speaking, the
handling of the NIC router became more tailored to its/our typical use cases.
Let's create a practical setup to explain the changes in detail. Assume that
there are two virtual subnets 192.168.1.0/24 and 192.168.2.0/24 within our
Genode system. They connect as Virtnet A and B to the router. The standard
gateway of the virtual networks is the NIC router with IP 192.168.*.1 . The
router's uplink, on the other hand, is connected to the NIC driver. It
interfaces the machine with our real-world home network 10.0.2.0/24. The home
network is connected to the internet through its standard gateway 10.0.2.1.
[image nic_router_basic]
The basic router configuration for this setup without any routing rules would
be as follows:
! <policy label_prefix="virtnet_a" domain="virtnet_a" />
! <policy label_prefix="virtnet_b" domain="virtnet_b" />
!
! <domain name="uplink" interface="10.0.2.55/24" gateway="10.0.2.1" />
! <domain name="virtnet_a" interface="192.168.1.1/24" />
! <domain name="virtnet_b" interface="192.168.2.1/24" />
The first thing to notice is the changed usage of the policy tag. Previously,
the policy label - normally solely designated to correlate sessions with
configuration domains - was misused also as unique peer identifier in the
routing rules. This approach disregarded advanced label-matching techniques
such as the 'label_prefix' used above. Now, the whole NIC-router-specific
enhancement of the policy tag moved to the new '<domain>' tag, leaving the
policy tag only with its original purpose to select policies. Note that even
if this modification gives the impression, the router is not yet capable of
handling multiple NIC sessions at one domain at a time.
In the domain tag, the 'interface' attribute replaces the old policy attribute
named 'src'. That means, it tells the router which IP identity to use when
talking as itself to the domain. But in addition to that, the 'interface'
attribute also defines which subnet this identity and the domain belong to.
This reflects a basic decision we made during the reworking process: The new
NIC router is aware of subnets. Sessions of the same subnet have the same
configuration domain. We came to this conclusion as it solves some fundamental
problems with the old version. First, the equivalence of domain and subnet
enables us to link a default gateway to a subnet by adding the 'gateway'
attribute to the domain tag. In our example, this is done in the uplink
domain. The 'gateway' attribute is optional for a domain and replaces the
former 'via' attributes of the different routing rules. It is more efficient
and natural to have this value set only once at the corresponding subnet than
having it scattered all over the routing rules of the remote domains as done
before. If a domain has no default gateway, it drops all packets with a
foreign recipient.
The second advantage of a domain being equivalent to a subnet is that handling
ARP broadcasts becomes easy. It can be excluded that such ARP broadcasts
concern sessions outside the source domain anymore. And as sessions in the
same domain are not distinguishable to the routing, the broadcast can be sent
to all of them without breaking any rules.
Now, let's enhance our example by some routing rules. One pretty complicated
thing to do with the old NIC router was port forwarding. You had to combine
different routing rules, explicitly enable the back routing at the remote
side, and take care that NAT was applied - a lot of opportunities for
mistakes. With the new version, it became easier. Let's assume we have an HTTP
server in Virtnet A and an NTP server in Virtnet B. We want the NIC router to
act as proxy for their services in our home network.
[image nic_router_servers]
In order to achieve this, the uplink domain must be enhanced by two rules:
! <policy label_prefix="virtnet_a" domain="virtnet_a" />
! <policy label_prefix="virtnet_b" domain="virtnet_b" />
!
! <domain name="uplink" interface="10.0.2.55/24" gateway="10.0.2.1" />
! <tcp-forward port="443" domain="virtnet_a" to="192.168.1.2" />
! <udp-forward port="123" domain="virtnet_b" to="192.168.2.2" />
! </domain>
!
! <domain name="virtnet_a" interface="192.168.1.1/24" />
! <domain name="virtnet_b" interface="192.168.2.1/24" />
The TCP forwarding rule for port 443 (HTTP+TLS/SSL) redirects to IP address
192.168.1.2 in Virtnet A and the UDP forwarding rule for port 123 (NTP)
redirects to IP address 192.168.2.2 in Virtnet B. The Virtnet domains remain
empty as the router keeps track of the redirected transfers and routes back
reply packets automatically. Also automatically, the router applies NAT for the
server as it is in the nature of port forwarding.
Next, we add some clients to Virtnet B that like to talk to our home network
and the internet. We want them to be hidden via NAT when they do so. For
internet communication, they shall furthermore be limited to HTTP+TLS/SSL and
IMAP+TLS/SSL.
[image nic_router_client]
This is what the router configuration looks now:
! <policy label_prefix="virtnet_a" domain="virtnet_a" />
! <policy label_prefix="virtnet_b" domain="virtnet_b" />
!
! <domain name="uplink" interface="10.0.2.55/24" gateway="10.0.2.1" />
! <tcp-forward port="443" domain="virtnet_a" to="192.168.1.2" />
! <udp-forward port="123" domain="virtnet_b" to="192.168.2.2" />
! <nat domain="virtnet_b" tcp-ports="1000" udp-ports="1000">
! </domain>
!
! <domain name="virtnet_a" interface="192.168.1.1/24" />
! <domain name="virtnet_b" interface="192.168.2.1/24" >
! <tcp dst="10.0.2.0/24"> <permit-any domain="uplink" /> </tcp>
! <udp dst="10.0.2.0/24"> <permit-any domain="uplink" /> </udp>
! <tcp dst="0.0.0.0/0">
! <permit port="443" domain="uplink" />
! <permit port="993" domain="uplink" />
! </tcp>
! </domain>
There are several new tag types. One of them is the NAT configuration for
Virtnet B in the uplink domain. In contrast to the former NIC-router version
where NAT settings were part of the source domain, NAT is now configured in
the target domain with a sub-tag for each source. This has the advantage
of supporting heterogeneous NAT configurations for a packet source depending
on which domain it talks to. Besides, it is more intuitive to read. Apart from
that, the NAT settings haven't changed.
Furthermore, there are the new TCP and UDP tags in the Virtnet-B domain. The
first two of them have a 'permit-any' sub-tag. With this combination, we open
all ports to IP addresses of the 10.0.2.0/24 subnet, our home network, and
route them to the uplink domain. TCP packets that don't match these first two
rules may fall back to the third. This TCP rule doesn't have all ports opened
but only 443 (HTTP+TLS/SSL) and 993 (IMAP+TLS/SSL). Both ports are again bound
to the uplink domain. As the IP filter 0.0.0.0/0 of the surrounding rule isn't
restrictive, we now also route packets to a foreign destination. The NIC
router redirects such packets to the default gateway of our home network.
Compared to the old router version where IP and UDP/TCP routing had to be
combined for this purpose, the new TCP and UDP rules with their
port-permission sub-rules have some notable advantages. Like port-forwarding
rules, TCP and UDP rules always imply link-state tracking in order to route
back reply packets automatically. This can be seen also in our example as no
further routing rules had to be added to the uplink domain. This aspect is
clear from the outermost rule and not dependent on sub-rules anymore.
Furthermore, the strict separation of UDP and TCP routing prevents
configuration faults and increases readability. Last but not least, the
'permit-any' rule allows something new. Opening all ports for an address range
was previously only possible without link-state tracking as it could be
expressed only on the IP level.
At this point, we have thoroughly discussed the layer-3 routing abilities of
the new NIC router and our focus has indeed moved more into this direction.
Even though IP routing is still available, we found that it should be more
clearly separated from the rest. To illustrate this feature, we enhance our
example again. We want the Virtnets to be allowed to communicate to each other
without any restrictions. For that purpose, we add two more rules to the
router configuration:
! <policy label_prefix="virtnet_a" domain="virtnet_a" />
! <policy label_prefix="virtnet_b" domain="virtnet_b" />
!
! <domain name="uplink" interface="10.0.2.55/24" gateway="10.0.2.1" />
! <tcp-forward port="443" domain="virtnet_a" to="192.168.1.2" />
! <udp-forward port="123" domain="virtnet_b" to="192.168.2.2" />
! <nat domain="virtnet_b" tcp-ports="1000" udp-ports="1000">
! </domain>
!
! <domain name="virtnet_a" interface="192.168.1.1/24" />
! <ip dst="192.168.2.0/24" domain="virtnet_b"/>
! </domain>
!
! <domain name="virtnet_b" interface="192.168.2.1/24" >
! <tcp dst="10.0.2.0/24"> <permit-any domain="uplink" /> </tcp>
! <udp dst="10.0.2.0/24"> <permit-any domain="uplink" /> </udp>
! <tcp dst="0.0.0.0/0">
! <permit port="443" domain="uplink" />
! <permit port="993" domain="uplink" />
! </tcp>
! <ip dst="192.168.1.0/24" domain="virtnet_a"/>
! </domain>
As you can see, each of the new IP rules in the Virtnet domains match the
addresses of the opposite subnet and route to the corresponding domain. As
mentioned, the new IP rules and UDP/TCP rules are not combined anymore to
clearly distinguish IP routing from layer-3 routing because this decision has
far-reaching effects. First, in contrast to UDP and TCP routing, IP routing is
stateless. Thus, for each IP routing rule one has to be sure to have a
back-routing rule at the remote domain or else bidirectional communication
won't happen. And second, NAT does not apply to IP-routed packets. So, if
you're not aware of such packets, you may unintentionally reveal information
about a private network.
For more details on the new NIC router, you may refer to the comprehensive
documentation in the _repos/os/src/server/nic_router/README_ file and the
basic NIC-router test at _libports/run/nic_router.run_ .
Base framework
##############
Improved RPC mechanism
======================
Since we introduced Genode's current API for synchronous RPCs in
[https://genode.org/documentation/release-notes/11.05#New_API_for_type-safe_inter-process_communication - version 11.05],
inter-component communication within Genode has become almost a child's play.
The RPC framework leverages the C++ type system and templates to a great
effect. In contrast to the traditional use of IDL compilers, the interaction
with RPC objects provided by other components is robust and natural because
no language boundaries need to be crossed.
Still, a few differences between RPC calls and regular function calls remain.
In particular, there exist a few restrictions with regard to the types of
RPC function arguments. Those types did not just need to be POD (plain old
data) types but they had to be default-constructible, too. Whereas the former
restriction still applies (non-POD objects that include references or
vtables cannot be used as arguments), the latter limitation has been lifted
now. Generally, non-default-constructible types are a way to attain
simpler code because the special case of an "invalid" object does not need
to be considered. I.e., values of such types can be kept as constants as
opposed to variables. If an object exists (as equivalent to successful
instantiation), it is valid. With the improved RPC mechanism, the RPC
framework does no longer stay in the way in this respect.
Thanks to Edgard Schmidt for this welcome contribution!
Unification and tightening of session labels
============================================
In Genode, each session requested by a client component is labeled according
to the components that intermediate the session request. The client can
optionally specify a label of choice along with the session request. Its
parent prefixes the client-provided label by a label of its own. If the
session request is further passed to the parent's parent, the grandparent
prepends its own label. This works recursively. Consequently, the final label
as seen by the server is the product of the labeling policies of all
components on the route of the session request.
The label is used for two purposes. First, the server uses the label as
a key for a server-side policy selection. E.g., depending on the session label
received by the disk-partition server, the server decides which partition to
hand out to the client. Second, the label is used by intermediate components
to take session-routing decisions. E.g., based on the label of a file-system
session request, a parent component may route the request to one of several
file-system servers.
Originally, Genode did not impose a specific way of how labels are formed.
It was up to each intermediate component to filter the label of a session
request in any way desired. However, in practice, this freedom remained unused
and the very simple successive prefixing of labels prevails in all our use
cases. Each intermediate node concatenates its own label in front of the label
supplied by the originator of the session request. The different parts of the
label are separated with the character sequence '" -> "'. Some corner cases
were handles specially for aesthetic reasons. For example, if a client
provided no label, the parent would skip the pending separator. That said,
since each intermediate component had to provide the labeling policy, not all
components were consistent in these respects. Since we found no use for
arbitrary labeling policies, we decided to make the only prominent way of
session labeling mandatory for all intermediate components. We thereby removed
the aesthetically motivated corner cases and possible ambiguities. I.e., with
the original policy, it was not possible to distinguish a unlabeled session
requested by a client from a labeled session requested by the client's parent.
As a consequence, the stricter labeling must now be considered wherever
a precise label was specified as a key for a session route or a server-side
policy selection. The simplest way to adapt those cases is to use a
'label_prefix' instead of the 'label' attribute. Alternatively, the
'label' attribute may used by appending '" -> "' (note the whitespace).
Transition to new framework API
===============================
Since we fundamentally revised Genode's API in
[http://genode.org/documentation/release-notes/16.05#The_great_API_renovation - version 16.05],
we gradually adapt our existing components. Given that Genode comes with
over 300 components, this is no small feat. But with 30 percent of the
components converted, we already made substantial progress.
In some respects, the conversion is actually nearly complete. In particular,
the move away from format-string-based text output to our new type-safe output
facility has been applied to almost all components now. The former 'PDBG'
macro that is quite useful for temporary debug messages has been replaced with
a new version that must be manually included via the _base/debug.h_ header
file. Like the regular log functions, the new PDBG facility uses the type-safe
text-output facility.
Minor API adjustments
---------------------
While applying Genode's new API, we refined the API in the following respects:
We added a dedicated 'String' constructor overload to better accommodate
string literals. This overload covers the common case for initializing a
string from a literal without employing the 'Output' mechanism. This way, such
strings can by constructed without calling virtual functions, which in turn
makes the 'String' usable during the self-relocation phase of the dynamic
linker.
Up till now, several Genode components still rely on the use of 'snprintf'
whenever strings must be assembled out of smaller pieces. As we like to shun
format strings from Genode altogether, we needed an alternative mechanism.
Since we introduced the new type-safe text-output facilities in Genode 16.05,
there is an obvious solution: Let the 'String' constructor accept an arbitrary
list of arguments, which are turned into their respective textual
representation and appear concatenated in the resulting string. Consequently,
strings can be assembled with the same flexibility as log output. For the
construction of 'String' objects from character buffers of a known size, the
'Cstring' utility can be used, which takes a 'char const *' and an optional
length as arguments.
Several low-level types received support for the new output facilities, e.g.,
'Xml_node' or the network-related headers in _os/net/_.
In anticipation of the forthcoming package-management infrastructure, we try
to unify Genode's executable binaries across kernels and architectures
wherever reasonable. Of course, the latter is not possible with respect to the
used instructions. But unifying symbol information is deemed worthwhile. For
this reason, we changed the 'Genode::size_t' type to be always defined as an
'unsigned' 'long'. This is in contrast to GCC's built-in '__SIZE_TYPE__',
which is defined as 'unsigned int' on 32-bit architectures but 'unsigned long'
on 64-bit architectures.
OS-level infrastructure and device drivers
##########################################
New timeout-handing API
=======================
The new timeout API offers tools for easily multiplexing a single time
source among different timeouts. In general, the time source can be
implemented individually but we expect that the most prominent use case will
be the multiplexing of timer sessions. Thus, the timeout library also provides
a convenience tool for this use case. A library-usage example can be found
under _os/src/test/timeout_. If you're interested in implementing
your own time source, you can find an example at _os/include/os/timer.h_ .
Support for smart cards
=======================
We ported the [http://pcsclite.alioth.debian.org/pcsclite.html - PC/SC Lite]
library to Genode, which provides a commonly used API for communicating with
smart cards. It supports USB smart card readers, using the
[http://pcsclite.alioth.debian.org/ccid.html - CCID] library as driver.
The CCID driver itself requires [http://libusb.info - libusb] to access the
USB device.
Vanilla PC/SC Lite is structured as a client-server architecture, consisting
of the 'pcscd' daemon, which runs on a privileged user account and manages all
card reader devices, and one or more non-privileged client applications, which
communicate with pcscd to access the card readers. On Genode, pcscd's role as
privileged device manager is not really needed, since the devices can also be
managed using Genode's configuration mechanisms. For this reason, we merged
the part of pcscd which implements the API with the pcsc-lite client library.
In the current state, a Genode application using PC/SC Lite can access a single
card reader device, which is selected using its USB product ID and vendor ID in
the application's configuration and in the policy of the USB driver.
More configuration details can be found in the README files of the PC/SC Lite,
CCID, and libusb libraries in the libports repository and in the accompanying
_smartcard.run_ script.
Libraries and applications
##########################
Time-based password generation
==============================
A time-based one-time password authentication client that adheres to the
Google Authenticator standard has been introduced into the
[https://github.com/genodelabs/genode-world - world repository].
Single use, time-based passwords are commonly used as an additional
authentication step for web-based services. In this scheme, a user generates
and presents a six digit passcode to a service generated using a shared secret
and a timestamp. This short passcode length makes manual entry convenient so
that the shared secret may be stored on a separate device than the service
client, such as a smartphone, layering the security properties of both
devices.
The 'gtotp' VFS plugin provides these passcodes by embedding the generator as
a special file in the file-system layer of a component. This approach provides
readily available passcodes for programmatic and manual use without enlarging
the code base to encompass a GUI, command-line, or networked interface.
At the time of this release, the common use case is to manually retrieve codes
for clients running in VirtualBox by reading special files with an isolated
instance of the Noux runtime. Storing the shared secret on the same device
contradicts the recommendations of the standard but the trade-off is that the
software stack required to host the shared secret is significantly smaller
than that found on a mobile device.
Random number generator testing
===============================
No random number generator can be proved to be good, but empirical statistical
tests can prove that some are bad. A port of the TestU01 RNG test suite is
provided in the world repository. The TestU01 batteries give independent
assurance of the fitness of Genode's CPU jitter based RNG and are available
for testing future physical and non-phyical RNGs.
VirtualBox on top on the NOVA hypervisor
########################################
Both VirtualBox-based virtual machine monitors on Genode got updated to the
latest revision as provided by Oracle, namely 4.3.40 and 5.1.10 - mainly to
stay close to the upstream versions.
Platforms
#########
Unified handling of boot modules
================================
Until now, the way of passing boot modules from the boot procedure to the core
component, which core provides as ROM modules, varied from platform to
platform. Either we used a multiboot-compliant bootloader that accepts
multiple modules, or the platform provided some specific way of linking binary
modules together with the kernel, e.g., the Elfweaver tool of OKL4.
By unifying the boot-module handover, we further reduce platform specific core
code. Thereby, maintenance costs are decreased, and code analysis becomes
easier. With this new solution, when issuing to build the core component:
! make core
within the build system, only a core library gets built. Not until all
binaries needed by a run-script are available, a final image is linked
together using the core library and all additional binaries. The core
component now can access its ROM modules directly via addresses contained in
its binary. As a side effect of this change, there is no core binary in the
'bin' or 'core' directory of the corresponding build directory available
anymore. Instead, you will find the core binary with no ROM modules, but
including debug information under 'var/run/*.core' within your build
directory. The concrete name depends on the name of the run-script.
The new approach is used on all platforms except Linux where the ROM modules
still need to be accessed via the file-system.
NOVA hypervisor
===============
We extended the kernel to support the asynchronous delegation of kernel
resources. Up to now, resources could only be delegated during RPC or during
the initial protection-domain construction. With this extension, the
construction and setup of new protection domains, threads, and especially
virtual CPUs for the VirtualBox VMM became more straightforward and several
quirks inside the 'core' component could be dropped. The added kernel syscall
expects the NOVA-kernel capabilities of the source and target protection
domains, which effectively renders the operation solely available to 'core' -
as only holder of the NOVA protection domain capabilities.
Additionally, we changed the CPU ID enumeration in Genode/NOVA to a
predictable order. The lower CPU IDs used via the Genode 'Cpu_session'
interface now correspond to the first hyper-thread of all physical CPU cores.
For example, on a quad-core machine with hyper-threading enabled Genode's CPU
IDs 0-3 refer to the first hyper-threads of all physical cores and IDs 4-7 to
the second hyper-threads.

1033
doc/release_notes-17-05.txt
View File

File diff suppressed because it is too large Load Diff

651
doc/release_notes-17-08.txt
View File

@@ -1,651 +0,0 @@
===============================================
Release notes for the Genode OS Framework 17.08
===============================================
Genode Labs
The flagship feature of Genode 17.08 has been in the works for more than a
year: The support for hardware-accelerated graphics on Intel Gen-8 GPUs. This
is an especially challenging topic because it is riddled with terminology,
involves highly complex software stacks, carries a twisted history with it,
and remains to be a moving target. It took up a lot of patience to build up a
profound understanding of the existing driver architectures and the mechanisms
offered by modern graphics hardware. On the other hand, with the proliferation
of hardware-based sandboxing features like virtual GPU memory and hardware
contexts, we found that now is the perfect time for a clean-slate design of a
microkernelized GPU driver.
Section [Hardware-accelerated graphics for Intel Gen-8 GPUs] introduces this
work, which includes our new GPU multiplexer as well as the integration with
the client-side Mesa protocol stack.
The second focus of the current release is the extension of Genode's supported
base platforms. Most prominently, we upgrade the seL4 kernel to version 6.0
while extending the architecture support from 32-bit x86 to ARM and 64-bit
x86 (Section [The seL4 kernel on ARM and 64-bit x86 hardware]). To bring
Genode closer to cloud-computing scenarios, we added basic support for
executing Genode scenarios as Xen DomU domains (Section [Genode as Xen DomU]).
Furthermore, the Muen separation kernel has been updated to a current version.
As a cross-kernel effort, there is work under way to boot Genode-based
systems via UEFI, currently addressing the NOVA, base-hw, and seL4 kernels.
Among the many other functional additions are a new VFS plugin for accessing
FAT file systems, new components like _sequence_ and _fs_report_ that aid new
system compositions, and our evolving custom package-management
infrastructure.
Hardware-accelerated graphics for Intel Gen-8 GPUs
##################################################
The ability to leverage hardware-accelerated graphics is generally taken for
granted in modern commodity operating systems. The user experience of
modern desktop environments, web-browser performance, and obviously games
depend on it. On the other hand, the benefit of hardware-accelerated graphics
comes at the expense of tremendous added complexity in the lower software
stack, in particular in system components that need to be ultimately trusted.
For example, with circa 100 thousand lines of code, the Intel GPU driver in
the Linux kernel is an order of magnitude more complex than a complete modern
microkernel. In a monolithic-kernel-based system, this complexity is
generally neglected because the kernel is complex anyway. But in
microkernel-based scenarios optimized for a trusted computing base in the
order of a few ten thousand lines of code, it becomes unacceptable.
Fortunately, recent generations of graphics hardware provide a number of
hardware features that promise to solve this conflict, which prompted us to
investigate the use of these features for Genode.
During this year's Hack'n'Hike event, we ported the ioquake3 engine to Genode.
As preliminary requirement, we had to resurrect OpenGL support in our aging
graphics stack and enable support for current Intel HD Graphics devices (IGD).
We started by updating Mesa from the old 7.8.x to a more recent 11.2.2 release.
Since we focused mainly on supporting Intel devices, we dropped support for the
Gallium back end as Intel still uses the old DRI infrastructure. This decision,
however, also influenced the choice of the software rendering back end. Rather
than retaining the softpipe implementation, we now use swrast. In addition, we
changed the available OpenGL implementation from OpenGL ES 2.x to the fully
fledged OpenGL 4.5 profile, including the corresponding shader language
version. As with the previous Mesa port, EGL serves as front end API for
system integration and loads a DRI back-end driver (i965 or swrast). EGL
always requests the back-end driver 'egl_drv.lib.so' in form of a shared
object. Genode's relabeling features are used to select the proper back end
via a route configuration. The following snippet illustrates such a
configuration for software rendering:
! <start name="gears" caps="200">
! <resource name="RAM" quantum="32M"/>"
! <route>
! <service name="ROM" label="egl_drv.lib.so">
! <parent label="egl_swrast.lib.so"/>
! </service>
! <any-service> <parent/> <any-child/> </any-service>
! </route>
! </start>
With the graphics-stack front end in place, it was time to take care of the
GPU driver. In our case this meant implementing the DRM interface in our
ported version of the Intel i915 DRM driver. Up to now, this driver was solely
used for mode setting while we completely omitted supporting the render
engine.
[image mesa_genode]
With this new and adapted software stack, we successfully could play ioquake3
on top of Genode with a reasonable performance in 1080p on a Thinkpad X250.
During this work, we gathered valuable insights into the architecture of a
modern 3D-graphics software stack as well as into recent Intel HD Graphics
hardware. We found that the Intel-specific Mesa driver itself is far more
complex than its kernel counter part. The DRM driver is mainly concerned with
resource and execution management whereas the Mesa driver programs the GPU.
For example, amongst others, Mesa compiles the OpenGL shaders into a
GPU-specific machine code that is passed on to the kernel for execution.
While inspecting the DRM driver, it became obvious that one of the reasons for
its complexity is the need to support a variety of different HD Graphics
generations as well as different features driven by software-usage patterns.
For our security related use cases, it is important to offer a clear isolation
and separation mechanism per client. Hardware features provided by modern
Intel GPUs like per-process graphics translation tables (PPGTT) and hardware
contexts that are unique for each client make it possible to fulfill these
requirements.
By focusing on this particular feature set and thus limiting the supported
hardware generations, the development of a maintainable GPU multiplexer for
Genode became feasible. After all, we strive to keep all Genode components as
low complex as possible, especially resource multiplexers like such a GPU
multiplexer.
[image intel_gpu_drv]
This image shows multiple GPU-session clients and the resources they are
using. The fence registers as well as the aperture is partitioned between
them, the PPGTT is backed by the system memory, and the contexts are located
in disjoint GGTT regions.
Within four months, we implemented an experimental GPU multiplexer for Intel
HD Graphics Gen8 (Broadwell class) devices. We started out defining a GPU
session interface that is sufficient to implement the API used by the DRM
library. For each session, the driver creates a context consisting of a
hardware context, a set of page tables (PPGTT), and a part of the aperture.
The client may use the session to allocate and map memory buffers used by the
GPU. Each buffer is always eagerly mapped 1:1 into the PPGTT by using the
local virtual address of the client. Special memory buffers like an image
buffer are additionally mapped through the aperture to make use of the
hardware-provided de-tiling mechanism. As is essential in Genode components,
the client must donate all resources that the driver might need to fulfill the
request, i.e., quota for memory and capability allocations. Clients may
request the execution of their workload by submitting an execution buffer. The
GPU multiplexer will then enqueue the request and schedule all pending
requests sequentially. Once the request is completed, the client is notified
via a completion signal.
[image multi_gl]
Example scenario of multiple OpenGL programs that use the new GPU multiplexer
for hardware-accelerated rendering.
We consider this first version of the GPU driver as experimental. As of now,
it only manages the render engine of the GPU. Mode-setting or rather display
handling must be performed by another component. Currently, the VESA driver is
used for this purpose. It also lacks any power-management functionality and
permanently keeps the GPU awake. Both limitations will be addressed in future
releases and support for Gen9+ (Skylake) and newer devices might be added.
In its current incarnation, the GPU multiplexer component consists of about
4,200 lines of code whereas the Mesa DRI i965 driver complements the driver at
the client side with about 78,000 lines of code.
The seL4 kernel on ARM and 64-bit x86 hardware
##############################################
With the 16.08 release, we brought the seL4 support to a level to be
considered being on par with the other supported kernels. At the time,
Genode's use of seL4 was limited to 32-bit x86 platforms.
In the current release, we extend the platform support to ARM and 64-bit x86.
We started this line of work with an incremental kernel upgrade from version
3.2.0 to 5.2.0 and finally to seL4 6.0. Through these upgrades, we were able
to drop several Genode-specific seL4 patches, which were required in the 16.08
release. One major improvement of version 6.0 compared to earlier versions is
the handling of device-memory announcements by the kernel to Genode's roottask
_core_.
With the kernel update in place, we inspected the x86-specific part thoroughly
while splitting and separating it properly into architecture-agnostic and
architecture-dependent parts. Upon this work, we added the
architecture-specific counterparts for x86_64 and ARM. One major work item was
to make the page-table handling in Genode's core aware and generic enough to
handle the different page-table sizes of the three architectures.
For the ARM support, we decided to enable the i.MX6 FreeScale based SoC,
namely the Wandboard Quad board. Since the seL4 kernel interface provides no
timeout support, we revived a user-level timer driver that we originally
developed for our custom base-hw kernel: The so-called EPIT timer, which is
part of most i.MX SoCs.
We finished the essential work for the mentioned three platforms in
less time than expected and, thereby, had spare time to address additional
features.
First, we enabled multiprocessor support for Genode/seL4 on x86 and
thread-priority support for all seL4 platforms. Additionally, we were able to
utilize the seL4 benchmark interface for Genode's trace infrastructure in
order to obtain utilization information about threads and CPUs. The Genode
components _top_ (text-based) and _cpu_load_monitor_ (graphical) are now
usable on Genode/seL4.
Finally, as we are currently exploring the support for booting various kernels
via UEFI on x86, we took the chance to investigate the steps needed to boot
seL4 via UEFI. UEFI firmware does not always provide a compatibility support
module (CSM) for legacy BIOS boot support. Hence, we extended the seL4 kernel
for Genode according to the Multiboot2 specification, which enables us to
start Genode/seL4 together with GRUB2 - as an UEFI capable bootloader - on
machines missing CSM support.
Base framework and OS-level infrastructure
##########################################
Simplified IOMMU handling
=========================
When IOMMUs are used on x86, all host memory targeted via direct memory
accesses (DMA) by devices must eagerly be registered in the respective I/O
page table of the device. Up to now, Genode supports IOMMUs on NOVA only. On
this kernel, a device protection domain is represented as a regular protection
domain with its virtual memory layout being used for both the CPU's MMU and
the device. Traditionally, mappings into such virtual memory spaces are
inserted on demand as responses to page faults. However, as there are no page
faults for DMA transactions, DMA buffers must always be eagerly mapped. The
so-called device PD hid this gap for NOVA. In anticipation of adding IOMMU
support for more kernels, we desired to generalize the device-PD mechanism by
introducing an explicit way to trigger the insertion of DMA memory into the
proper page tables.
We extended the PD-session interface by a 'map' function, which takes a
virtual memory region of the PD's virtual address space as argument. The page
frames of the previously attached dataspaces are added eagerly by core to the
IOMMU page-tables. With this explicit 'map' support, we were able to replace
the Genode/NOVA-specific device-PD implementation with a generic one, which
will easily accommodate other kernels in the future.
New report server for capturing reports to files
================================================
The report session is a simple mechanism for components to publish structured
data without the complexity of a file-system layer. In the simplest case, a
client component will produce a report and communicate it directly to a
component acting as a server. The disadvantage is that the report client
becomes reliant on the liveliness and presence of the consumer component. So
in the more robust case, the _report_rom_ component acts as the server hosting
the report service as well as a ROM service for components consuming reports.
The _report_rom_ server permits ROM access only to clients matching an
explicit configuration policy. This is good for security but opaque to a user.
Reports can only be read where an explicit policy is in place and only a
single report session can report to an active ROM session.
The new _fs_report_ component is a friendlier and more flexible report server.
Reports are written to a file system using a file and directory hierarchy that
expresses session routing. This allows for intuitive report inspection and
injection via a file system. When used with the _ram_fs_ and _fs_rom_ servers,
it can also replicate the functionality of _report_rom_.
New runtime environment for starting components sequentially
============================================================
The _init_ component is a prime example of software with an emphasis on
function over features. It is the fundamental building block for combining
components yet its behavior is simple and without heuristics. Like other
contemporary init managers, it starts components in parallel, but to a more
extreme degree in that it has no concept of "runlevels" or "targets", all
components are started as soon as possible. The concrete sequence of execution
is instead determined by when server components make service announcements and
how quickly they respond to client requests.
In some cases, the execution of one component must not occur until the
execution of another component ends, be it that the first produces output that
is consumed by the second, or that the two contend for a service that cannot
be multiplexed. Init contains no provisions to enforce ordering. But we are
free to define new behaviors in other management components.
The solution to the problem of ordering is the _sequence_ component. Sequence
walks a list of children and executes them in order, one at a time. With only
one child active, there is no need for any local resource or routing
management. By applying the same session label transformations as init,
external routing and policy handling are unchanged.
An example of ordering a producer and consumer within an init configuration
follows:
! <start name="sequence">
! <resource name="RAM" quantum="128M"/>
! <config>
! <start name="producer">
! <config .. />
! </start>
! <start name="consumer">
! <config .. />
! </start>
! </config>
! <route>
! <service name="LOG" label_prefix="producer">
! <child name="log_a"/> </service>
! <service name="LOG" label_prefix="consumer">
! <child name="log_b"/> </service>
! <any-service> <parent/> <any-child/> </any-service>
! </route>
! </start>
Support for boot-time initialized frame buffer
==============================================
UEFI-based systems do not carry along legacy BIOS infrastructure, on which
our generic VESA driver depends. Hence, when booting via UEFI, one has to use
either a hardware-specific driver like our Intel-FB driver or - alternatively -
facilitate generic UEFI mechanisms.
Instead of booting in VGA text mode and leaving the switch to a graphics mode
(via real-mode SVGA BIOS subroutines) to the booted OS, UEFI employs the
so-called graphics output protocol as a means to setup a reasonable default
graphics mode prior booting the operating system. In order to produce
graphical output, the operating system merely has to know the physical address
and layout of the frame buffer. Genode's core exposes this information as the
_platform_info_ ROM module. The new _fb_boot_drv_ driver picks up this
information to provide a Genode framebuffer session interface. Hence, on
UEFI-based systems, it can be used as a drop-in replacement for the VESA
driver. In contrast to the VESA driver, however, it is not able to switch
graphics modes at runtime.
The new component is located at _os/src/drivers/framebuffer/boot/_. Thanks
to Johannes Kliemann for this contribution.
Extended non-blocking operation of the VFS
==========================================
In
[https://genode.org/documentation/release-notes/17.02#VFS_support_for_asynchronous_I_O_and_reconfiguration - version 17.02],
we added support for non-blocking reads from the VFS in the form of the
'read_ready()', 'queue_read()', and 'complete_read()' functions. Since then,
it has become obvious that blocking within the VFS is not only problematic in
the VFS server itself when multiple clients are connected, but also when the
VFS is deployed in a multi-threaded environment and a VFS plugin needs to
reliably wait for I/O-completion signals.
For this reason, we reworked the interface of the VFS even more towards
non-blocking operation and adapted the existing users of the VFS accordingly.
The most important changes are:
* Directories are now created and opened with the 'opendir()' function and
the directory entries are read with the 'queue_read()' and 'complete_read()'
functions.
* Symbolic links are now created and opened with the 'openlink()' function and
the link target is read with the 'queue_read()' and 'complete_read()'
functions and written with the 'write()' function.
* The 'write()' function does not wait for signals anymore. This can have the
effect that data written by a VFS library user has not been processed by a
file-system server when the library user asks for the size of the file or
closes it (both done with RPC functions at the file-system server). For this
reason, a user of the VFS library should request synchronization before
calling 'stat()' or 'close()'. To make sure that a file-system server has
processed all write request packets that a client submitted prior the
synchronization request, synchronization is now requested at the file-system
server with a synchronization packet instead of an RPC function. Because of
this change, the synchronization interface of the VFS library has been split
into the 'queue_sync()' and 'complete_sync()' functions.
Making block sessions read-only by default
==========================================
Genode server components are expected to apply the safest and strictest
behavior when exposing cross-component state or persistent data. In practice
block and file-system servers only allow access to clients with explicitly
configured local policies. The file-system servers enforce an additional
provision that sessions are implicitly read-only unless overridden by policy.
This release introduces a similar restriction to the AHCI driver and partition
multiplexer. Clients of these servers require an affirmative 'writeable'
attribute on policies to permit the writing of blocks. Write permission at
these servers may also be revoked by components that forward block-session
requests by placing 'writeable="no"' into session-request arguments.
All users of _ahci_drv_ and _part_blk_ are advised that this change may break
existing configurations without explicit 'writeable' policies.
Refined time handling
=====================
Release 17.05 introduced a
[https://genode.org/documentation/release-notes/17.05#New_API_for_user-level_timing - new API for user-level timing]
named _timeout framework_. Together with this new framework came a
comprehensive test that stresses all aspects of the interface. During the past
few months, this test has turned out to be an enrichment for Genode far beyond
its original scope. As the test significantly raised the standards in
user-level timing, it also sharpened our view on the measurement precision of
various timer drivers and timestamps, which act as input for the framework.
This revealed several problems previously unidentified. For instance, we
improved the accuracy and stability of the time values provided by the drivers
for the Raspberry-Pi timer, the Cortex-A9 timer, the PIT, and the LAPIC. We
also were able to further optimize the calibration of the TSC in the NOVA
kernel.
Additionally, the test also helped us to refine the timeout framework itself.
The initial calibration of the framework - that previously took about 1.5
seconds - is now performed much quicker. This makes microseconds-precise time
available immediately after the timer connection switched to the modern
fine-grained mode of operation, which is a prerequisite for hardware drivers
that need such precision during their early initialization phase. The
calculations inside the framework also became more flexible to better fit the
characteristics of all the hardware and kernels Genode supports.
Finally, we were able to extend the application of the timeout framework. Most
notably, our C runtime uses it as timing source to the benefit of all
libc-using components. Another noteworthy case is the USB driver on the
Raspberry Pi. It previously couldn't rely on the default Genode timing but
required a local hardware timer to reach the precision that the host
controller expected from software. With the timeout framework, this workaround
could be removed from the driver.
FatFS-based VFS plugin
======================
Genode has supported VFAT file-systems since the 9.11 release when the
[http://elm-chan.org/fsw/ff/00index_e.html - FatFS] library was first ported.
The 11.08 release fit the library into the libc plugin architecture and
in 12.08 FatFS was used in the _ffat_fs_ file-system server. Now, the 17.08
release revisits FatFS to mold the library into the newer and more flexible
VFS plugin system. The _vfs_fatfs_ plugin may be fitted into the VFS server or
used directly by arbitrary components linked to the VFS library. As the
collection of VFS plugins in combination with the VFS file-system server has a
lower net maintenance cost than multiple file-system servers, the _ffat_fs_
server will be retired in a future release.
Enhanced GUI primitives
=======================
Even though we consider Qt5 as the go-to solution for creating advanced
graphical user interfaces on top of Genode, we also continue to explore an
alternative approach that facilitates Genode's component architecture to an
extreme degree. The so-called menu-view component takes an XML description of
a dialog as input and produces rendered pixels as output. It also gives
feedback to user input such as the hovered widget at a given pointer position.
The menu view does not implement any application logic but is meant to be
embedded as a child component into the actual application. This approach
relieves the application from the complexity (and potential bugs) of widget
rendering. It also reinforces a rigid separation of a view and its underlying
data model.
The menu view was first introduced in
[https://genode.org/documentation/release-notes/14.11#New_menu_view_application - version 14.11].
The current release improves it in the following ways:
* The new '<float>' widget aligns a child widget within a
larger parent widget by specifying the boolean attributes 'north', 'south',
'east', and 'west'. If none is specified, the child is centered. If opposite
attributes are specified, the child is stretched.
* A new '<depgraph>' widget arranges child widgets in the form of a
dependency graph, which will be the cornerstone for Genode's upcoming
interactive component-composition feature. As a prerequisite for
implementing the depgraph widget, Genode's set of basic graphical primitives
received new operations for drawing sub-pixel-accurate anti-aliased lines
and bezier curves.
* All geometric changes of the widget layout are animated now. This includes
structural changes of the new '<depgraph>' widget.
[image depgraph]
The menu-view component is illustrated by the run script at
_gems/run/menu_view.run_.
C runtime
=========
The growing number of ported applications used on Genode is accompanied by the
requirement of extensive POSIX compatibility of our C runtime. Therefore, we
enhanced our implementation by half a dozen features (e.g., O_ACCMODE
tracking) during the past release cycle. We thank the contributors of patches
and test cases and will continue our efforts to accommodate more ported
open-source components in the future.
Libraries and applications
##########################
Mesa adjustments
================
The Mesa update required the adaption of all components that use OpenGL.
In particular that means the Qt5 framework. Furthermore, we also enabled
OpenGL support in our SDL1 port.
As playground, there are a few OpenGL examples. The demos are located under
_repos/libports/src/test/mesa_demos_, which use the EGLUT bindings. There
are also some SDL based examples in the world repository under
_repos/world/src/test/sdl_opengl_.
Package management
==================
The previous release featured the initial version of Genode's
[https://genode.org/documentation/release-notes/17.05#Package_management - custom package-management tools].
Since then, we continued this line of work in three directions.
First, we refined the depot tools and the integration of the depot with our
custom work-flow ("run") tool. One important refinement is a simplification of
the depot's directory layout for library binaries. We found that the initial
version implied unwelcome complexities down the road. Instead of placing
library binaries in a directory named after their API, they are now placed
directly in the architecture directory along with regular binaries.
Second, driven by the proliferated use of the depot by more and more run
scripts, we enhanced the depot with new depot recipes as needed.
Third, we took the first steps to use the depot on-target. The experimentation
with on-target depots is eased by the new 'create_tar_from_depot_binaries'
function of the run tool, which allows one to assemble a new depot in the form
of a tar archive out of a subset of packages. Furthermore, the new
_depot_query_ component is able to scan an on-target depot for runtime
descriptions and returns all the information needed to start a subsystem based
on the depot content. The concept is exemplified by the new
_gems/run/depot_deploy.run_ script, which executes the "fs_report" test case
supplied via a depot package.
Platforms
#########
Genode as Xen DomU
==================
We want to widen the application scope of Genode by enabling users to easily
deploy Genode scenarios on Xen-based cloud platforms.
As a first step towards this goal, we enhanced our run tool to support running
Genode scenarios as a local Xen DomU, managed from within the Genode build
system on Linux running as Xen Dom0.
The Xen DomU runs in HVM mode (full virtualization) and loads Genode from an
ISO image. Serial log output is printed to the text console and graphical
output is shown in an SDL window.
To use this new target platform, the following run options should be defined in
the 'build/x86_*/etc/build.conf' file:
! RUN_OPT = --include boot_dir/$(KERNEL)
! RUN_OPT += --include image/iso
! RUN_OPT += --include power_on/xen
! RUN_OPT += --include log/xen
! RUN_OPT += --include power_off/xen
The Xen DomU is managed using the 'xl' command line tool and it is possible to
add configuration options in the 'xen_args' variable of a run script. Common
options are:
* Disabling the graphical output:
! append xen_args { sdl="0" }
* Configuring a network device:
! append xen_args { vif=\["model=e1000,mac=02:00:00:00:01:01,bridge=xenbr0"\] }
* Configuring USB input devices:
! append xen_args { usbdevice=\["mouse","keyboard"\] }
Note that the 'xl' tool requires super-user permissions. Interactive
password input can be complicated in combination with 'expect' and is not
practical for automated tests. For this reason, the current implementation
assumes that no password input is needed when running 'sudo xl', which can
be achieved by creating a file '/etc/sudoers.d/xl' with the content
! user ALL=(root) NOPASSWD: /usr/sbin/xl
where 'user' is the Linux user name.
Execution on bare hardware (base-hw)
====================================
UEFI support
------------
Analogously to our work on the seL4 and NOVA kernels in this release, we
extended our base-hw kernel to become a Multiboot2 compliant kernel. When used
together with GRUB2, it can be started on x86 UEFI machines missing legacy
BIOS support (i.e., CSM).
RISC-V
------
With Genode version 17.05, we updated base-hw's RISC-V support to privileged
ISA revision 1.9.1. Unfortunately, this implied that dynamic linking was not
supported on the RISC-V architecture anymore. Since dynamic linking is now
required for almost all Genode applications by default, this became a severe
limitation. Therefore, we revisited our RISC-V implementation - in particular
the kernel entry code - to lift the limitation of being able to execute only
statically linked binaries.
Additionally, we integrated the Berkeley Boot Loader (BBL), which bootstraps
the system and implements the machine mode, more closely into our build
infrastructure. We also added a new timer implementation to base-hw by using
the _set timeout SBI_ call of BBL.
What still remains missing is proper FPU support. While we are building the
Genode tool chain with soft float support, we still encounter occasions where
FPU code is generated, which in turn triggers compile time errors. We will
have to investigate this behavior more thoroughly, but ultimately we want to
add FPU support for RISC-V to our kernel and enable hardware floating point in
the tool chain.
Muen separation kernel
======================
Besides updating the Muen port to the latest kernel version as of end of June,
Muen has been added to Genode's automated testing infrastructure. This
includes the revived support for VirtualBox 4 on top of this kernel.
NOVA microhypervisor
====================
The current release extends NOVA to become a Multiboot2 compliant kernel. Used
together with GRUB2, NOVA can now be started on x86 UEFI machines missing
legacy BIOS support (called CSM).
GRUB2 provides the initial ACPI RSDP (Root System Description Pointer) to a
Multiboot2 kernel. The RSDP contains vital information required to bootstrap
the kernel and the operating system in general on today's x86 machines. To
make this information available to the user-level ACPI and ACPICA drivers, the
kernel propagates the RSDP to Genode's core, which - in turn - exposes it to
the user land as part of the _platform_info_ ROM module.
In order to ease the setup of an UEFI bootable image, we added a new image
module to our run-tool infrastructure. The run option 'image/uefi' can be used
instead of 'image/iso' in order to create a raw image that contains a EFI
system partition in a GUID partition table (GPT). The image is equipped by the
new 'image/uefi' module with the GRUB2 boot loader, a GRUB2 configuration, and
the corresponding Genode run scenario. The final image can be copied with 'dd'
to a bootable USB stick. Additionally, we added support to boot such an image
on Qemu leveraging [http://www.tianocore.org - TianoCore's] UEFI firmware.
As a side project, minor virtualization support for AMD has been added to
Virtualbox 4 and to the NOVA kernel on Genode. This enables us to run a 32-bit
Windows 7 VM on a 32-bit Genode/NOVA host on an (oldish) AMD Phenom II X4 test
machine.

898
doc/release_notes-18-02.txt
View File

@@ -1,898 +0,0 @@
===============================================
Release notes for the Genode OS Framework 18.02
===============================================
Genode Labs
After being developed for over a decade, Genode remained a mystery for many
people who looked at the project from a distance as it does not seem to fit
any established category of software. In 2018 - declared as the Year of Sculpt
on our [https://genode.org/about/road-map - roadmap] - this will hopefully
change. Genode 18.02 features the first revision of Sculpt, which is a
Genode-based operating system for general-purpose computing. After being used
as day-to-day OS by the entire team of Genode Labs for several months, we feel
that the time is right to share the system with a broader audience (Section
[Sculpt for Early Adopters]).
One fundamental feature of Sculpt is the ability to install and deploy
software from within the running operating system, which is universally
expected from any modern general-purpose OS. Section
[On-target package installation and deployment] presents Genode's unique
take on the topic of software installation and deployment.
Besides Sculpt, the current release has no shortage of other improvements.
Genode's growing arsenal of 3rd-party software received profound updates and
additions, including VirtualBox, Muen, seL4, several GNU packages, and
libraries. Also the user-level networking stack - including the Linux-based
LxIP stack and our custom NIC-router component - received a lot of attention.
Thanks to the added network driver for i.MX-based hardware, this networking
infrastructure becomes usable on embedded platforms based on this SoC.
Furthermore, the current release continues the cultivation of the Nim
programming language for Genode components.
Sculpt for Early Adopters
#########################
The current release features the first revision of Sculpt, which is a
Genode-based operating system for general-purpose computing. This initial
version is called Sculpt for Early Adopters (EA). Its target audience are
enthusiasts who are already familiar with Genode and are eager to use a
Genode-based operating system on their machines. As outlined on the
[https://genode.org/about/road-map - roadmap], later versions will become
increasingly approachable.
[image sculpt_overview]
Please refer to the official
[https://genode.org/documentation/articles/sculpt-ea - Sculpt documentation]
to step right into the adventure.
On-target package installation and deployment
#############################################
In May last year, we introduced the package-management concept for Genode to
pursue two goals. First, to overcome the naturally limited scalability of
composing Genode systems solely from source. This limit became evident in
complex system scenarios that incorporate a huge amount of 3rd party software.
Thanks to the introduced _depot_ concept and its integration in Genode's
workflow - in particular the run tool - the work of system integration became
much more structured (by caring about packages instead of individual build
targets), robust (by avoiding conditions in run scripts), and quick (by the
accelerated test cycle when using pre-built packages).
The second goal is the ability to update and extend a running Genode system on
the fly. We are happy to have reached this goal with the current release. As
exemplified by the Sculpt scenario, packages cannot only be used as building
blocks for system images but also as subsystems dynamically installed and
deployed on target. Even though installation and deployment are closely
related topics, both involve distinct challenges, which allow Genode to shine.
Installation / update
=====================
In traditional operating systems, the installation and update of system
software is the job of privileged programs. For example, a package manager in
a GNU/Linux system is typically executed with root privileges. This is
troublesome because the functionality of such a program is extremely complex.
In particular it is exposed to the network and has to parse content
originating from potentially untrusted parties. Therefore, potential software
vulnerabilities should be expected. However, in modern OSes, these programs
are just assumed to behave correctly. If this overly optimistic assumption
doesn't hold, the entire system is at risk.
Genode helps us to mitigate this problem by modelling each installation step
as a distinct component composition where each component has a well-defined
and extremely narrow role. The installation is an iterative sequence that
is orchestrated by the so-called download-manager component
(Figure [depot_download]).
[image depot_download]
Initially, the download manager receives a list of content to be installed
into the local depot, which is stored on the file system. The depot may
already be populated with (portions of) this content. In the first step, the
download manager must determine the parts that are missing. To do that, it
does not access the file system directly but instead hands over this task to a
disposable helper component called _depot-query_ that is spawned within a
dynamic init instance. This indirection has two benefits. First, the download
manager is not bothered with the complexity of accessing the file system. It
does not even have any notion of files. Second, the download manager is
effectively shielded from the file system. Should the file system misbehave,
the liveliness of the download manager remains unaffected.
[image depot_download_query_deps]
The depot-query component reports its findings to a report session. The report
eventually reaches the download manager as an updated ROM module. Given the
list of missing content, the download manager has to determine the information
of where to obtain the content from and the public key of the content creator.
This information is contained within the depot. So the download manager issues
another request to the depot-query component in order to obtain it.
[image depot_download_query_url]
Once the depot-query component has responded, the download manager knows what
content to get, where to get it, and how to verify it. To download the
content, it changes the dynamic init instance as follows.
[image depot_download_fetch]
The depot-query component is now gone. Actually, the entire depot has moved
out of sight. Instead, a fresh _fetchurl_ component is spawned. This component
is connected to the network as well as the writeable download directory
_public/_. Internally, fetchurl employs a complex software stack, which
includes the C runtime, curl, libssl, and libssh. Hence, we expect this
component to be vulnerable. Since it is facing the network, we assume that
vulnerabilities are exploitable. In the worst case where the component is
completely in the hands of an attacker, it may write wrong content into the
_public/_ location. But compared to executing curl or wget as root on a
traditional Unix system, the reach of an attack is quite limited. For example,
the mere existence of the download manager remains completely out of view of
fetchurl. However, the content of _public/_ must not be trusted. To reinforce
trust in the downloaded content, the content is accompanied with cryptographic
signatures created by the content creator. Before we touch the content, we
first check its authenticity. To perform this verification step, the download
manager reshapes the dynamic init instance as follows.
[image depot_download_verify]
Note that fetchurl exists no more and network connectivity is cut, effectively
disposing any form of malware that might have infected fetchurl. Next a new
_verify_ component enters the picture. It is configured with a list of content
to check, the signatures of the content, and the public key of the content's
presumed creator. Since it accesses the _public/_ location exclusively, it is
not prone to any potential time-of-check to time-of-use problems during the
verification. Under the hood, the _verify_ component employs a hugely complex
implementation based on GnuPG. It would be naive to fully trust this code.
However, when embedded in our scenario, the reach of a bug is limited because
the verify component has no access to any mutable system state. It could
merely give the wrong answer (which is of course bad but there is no way we
can magically solve this).
Knowing that the downloaded content is indeed the same content as intended
by the creator, it is time for extraction. For this step, the download
manager - again - reshapes the dynamic init instance:
[image depot_download_extract]
This time, both the _public/_ location as well as the trusted _depot/_ are
visible and a new _extract_ component is spawned. As the depot may host
content from multiple sources, which potentially distrust each other, the
content of each content provider resides in a dedicated subdirectory within
the depot. Instead of handing over access to the entire depot to the extract
tool, we mediate the file-system access via a _chroot_ component that limits
the view to the depot-provider's respective subdirectory. In the worst case
where a misbehaving content provider delivers a forged (but correctly signed)
archive to exploit a vulnerability of the extract component, the reach of the
attack remains limited to the content provider's space within the depot.
After the extraction step has completed, the depot is populated with the new
content, which may - in turn - include new dependency information. At this
point, the download manager starts a new iteration. This iterative process
terminates as soon as the depot-query component signals that no content of
the software installation is missing.
The bottom line here is that we are able to use complex and useful software
like curl, libarchive, liblzma, and GnuPG while largely distrusting it. In
contrast to this software that sums up to hundreds of thousand lines of code,
the download manager comprises less than 1000 lines of code. The software
installation procedure described above is implemented by the 'depot_download'
subsystem hosted in the gems repository and illustrated by an equally named
run script. It also forms the basis of the install/update mechanism of the
Sculpt scenario.
Deployment
==========
Once software has entered the system in the form of depot content, the
remaining question is how to turn this content into running subsystems. The
answer is given by the following illustration.
[image sculpt_deploy_runtime]
Like for the installation process described above, the scenario employs a
dynamic init instance that is accompanied by an orchestrating component. The
latter is called _depot-deploy_. The depot-deploy component queries
information from the depot using the same depot-query component that was used
during the installation. Based on the returned _blueprint_ information for the
to-be-deployed subsystems, it generates the configuration for the dynamic init
instance. The subsystems hosted within this init instance access the depot
content via mere ROM sessions as provided by the FS-ROM component. This makes
the use of the depot transparent to the hosted subsystems.
The depot-deploy component is located in the gems repository and accompanied
by a same-named run script. More importantly, it is featured in the deploy
runtime of the Sculpt system.
Base framework and OS-level infrastructure
##########################################
Increased default warning level
===============================
For building Genode components written in C++, the compiler flags -Wextra,
-Weffc++, and -Werror are now enabled in addition to -Wall by default.
If this strict warning level is inapplicable for a given component or
library, it is possible to explicitly disable the strictness in the
respective build-description file by adding the following line:
! CC_CXX_WARN_STRICT =
We adjusted almost all the code of the base, base-<kernel>, os, and demo
repositories to comply with this new warning level. For most components
hosted in the higher-level repositories (libports, ports, dde_*, gems),
the strictness is disabled as of now and will be enabled component-wise
wherever feasible.
While adjusting our code base, we identified the following patterns worth
mentioning:
* A class with virtual functions can no longer publicly inherit base
classes without a vtable. The inherited object may either be moved
to a member variable, or inherited privately. The latter would be
used for classes that inherit 'List::Element' or 'Avl_node'. In order
to enable the 'List' and 'Avl_tree' to access the meta data, the
'List' must become a friend.
* Instead of adding a virtual destructor to abstract base classes,
we inherit the new 'Interface' class, which contains a virtual
destructor. This way, single-line abstract base classes can stay
as compact as they are. The 'Interface' utility resides in
_base/include/util/interface.h_.
* With the new warning level, all member variables must be explicitly
initialized. Basic types may be initialized with '='. All other types
are initialized with braces '{ ... }' or as class initializers. If
basic types and non-basic types appear in a row, it is nice to only
use the brace syntax (also for basic types) and align the braces.
* If a class contains pointers as members, it must now also provide a
copy constructor and assignment operator. In most cases, one
would make them private, effectively disallowing the objects to be
copied. Unfortunately, this warning cannot be fixed by inheriting
our existing 'Noncopyable' class (the compiler fails to detect that
the inheriting class cannot be copied and still gives the error).
For now, we have to manually add declarations for both the copy
constructor and the assignment operator as private class members.
Those declarations should be prepended with a comment like this:
! /*
! * Noncopyable
! */
! Thread(Thread const &);
! Thread &operator = (Thread const &);
In the future, we plan to revisit these occurrences and try to replace
the pointers with references. In the presence of at least one
reference member, the compiler would no longer implicitly generate
a copy constructor. So we could remove the manual declaration.
The following caveats are expected, even if you disable the strictness
in your component:
* If your component has a class called 'Interface', it may collide with
the new 'Genode::Interface' class. You may have to disambiguate the
names.
* The 'Genode::Rpc_client' is no longer a 'Genode::Capability'. Hence,
classes inherited from 'Genode::Rpc_client' cannot refer to a
'Capability' but must refer to 'Genode::Capability'.
* The 'Surface' class is no longer copyable, which led to API
changes of users of this class. E.g., the 'Nitpicker_buffer'
utility does no longer offer accessors for the contained surfaces
but a new 'apply_to_surface' method that takes a lambda function as
argument.
Init
====
Init selects session routes based on the requested service and the client's
label. The latter can be matched as 'label' (exact match), 'label_prefix', or
'label_suffix' (either end of the label matches). With the new version, these
options are complemented with an additional 'label_last' attribute that covers
the prominent case where the last part of the label identifies a requested
resource at the server. A typical example is the routing of a ROM session
based on the name of the requested ROM module.
Reflecting the core log to the application level
================================================
Core records now log messages in a ring buffer and exports this
memory as ROM named 'core_log'. User applications may monitor this ring buffer
and present or transfer the content as appropriate. The example component in
_repos/os/src/app/log_core_ transforms the content into normal log
messages, which may be routed to graphical terminals or stored on
file systems, e.g. by using the fs_log server.
NIC-router improvements
=======================
During the past three months, the NIC router has received several improvements
that were mainly inspired by our daily experience with the component as part
of our Sculpt based working environments.
The most notable new feature is the support for multiple NIC sessions at one
domain. If multiple NIC-session clients connect to one domain, the NIC router
acts as a simple hub between them. I.e., for every packet that is routed to
the domain, each connected session receives a copy of the packet. The same
applies for domain-local packets, meaning packets that target an IP address
inside the IP subnet of the domain they came from. This domain-local
forwarding applies before considering any other routing rules. So, in other
words, it is not possible to route such traffic to another domain.
Furthermore, the logging features of the NIC router were improved. First, the
router is now capable of periodically sending a report via Genode's report
session. This can be activated by adding the new '<report>' node to the router
configuration:
! <config>
! <report interval_sec="5" bytes="yes" config="yes">
! ...
! </config>
So far, the report provides per-domain information about the amount of sent
and received data ('bytes' attribute) and the current IPv4 configuration like
IP address, subnet mask, and gateway address ('config' attribute).
Second, there is a new verbosity option in the '<config>' node:
! <config verbose_domain_state="yes">
When this option is set, the NIC router will output a short message to the log
for each general state change of a domain. Currently, this includes the
IP-configuration state (IP address, subnet mask, gateway address) and the
number of connected NIC sessions. This is a useful addition because the
purpose of the regular verbose option is to give a very deep insight into
almost every activity of the router, which is vital for debugging
sophisticated problems but normally floods the log. Therefore, the regular
verbose option is not viable for complex setups like a Sculpt desktop
environment. In such a context, the new domain-state verbosity is pretty
discreet but already gives a good hint on why, for instance, packets get
dropped despite the routing rules being correct.
Last but not least, the timeout configuration of the NIC router has been
reworked and now allows for a much more precise adaption to the network
environment. The former 'rtt_sec' attribute of the '<config>' node has been
replaced by the following new attributes (default values shown):
! <config dhcp_discover_timeout_sec="10"
! dhcp_request_timeout_sec="10"
! dhcp_offer_timeout_sec="10"
! udp_idle_timeout_sec="30"
! tcp_idle_timeout_sec="600"
! tcp_max_segm_lifetime_sec="30">
Details about the new attributes can be found in the
_os/src/server/nic_router/README_ file. The default values should be
appropriate for the common use case so that specifying them is normally not
necessary.
New watch mechanism for file-system session
===========================================
The file-system session already provided a way for watching files or
directories for changes. However, the original mechanism was arguably hard to
use. In addition to opening the to-be-watched file-system node, the client had
to submit a so-called content-changed request into the session's request
queue. In turn, the server delivered the change notification by acknowledging
this request.
The new mechanism is much less bureaucratic. A file or directory can be
watched by opening a watch handle rather than submitting a 'CONTENT_CHANGED'
packet to the server. Whenever a change happens at a node with an open watch
handle, a CONTENT_CHANGED packet will be sent from the server to the client.
This serializes the registration with other handle operations and separates
I/O handle state from notification handle state.
C runtime
=========
We changed libc's handling of 'clock_gettime' to be explicitly configurable
rather than relying on built-in heuristics. With the new version, the libc
opens a timer session as a time source only if the 'rtc' attribute of the
'<libc>' configuration node is defined. If not configured, 'clock_gettime'
returns 0.
This change may require the adjustment of components that implicitly rely on
the libc as time source. To enable such a component to use relative time
(based on a timer session) but no wall-clock time, one can manually provide a
pseudo real-time clock value as follows:
! <vfs>
! <dir name="dev">
! <log/> <null/> <inline name="rtc">2000-01-01 00:00</inline>
! </dir>
! </vfs>
! <libc stdout="/dev/log" stderr="/dev/log" rtc="/dev/rtc"/>
GUI stack and terminal improvements
===================================
Nit-FB improvements
-------------------
The nit_fb component provides a framebuffer and input service while using the
nitpicker GUI server as back end. The new version adds the 'initial_width' and
'initial_height' attributes, which accommodate the use case where nit_fb is
used in a dynamic fashion like as a client of a window system. Here, the
initial dimensions define the initial window size but - in contrast to the
existing 'width' and 'height' attributes - the actual size can change
afterwards.
Terminal resizing
-----------------
The terminal-session interface gained the ability to propagate resize events
from the server to the client. The new version of the graphical terminal uses
this mechanism to support window resizing as well as dynamically changing the
font size. At the client side, noux has become able to reflect terminal-size
changes to noux applications. Applications based on ncurses (e.g., vim) are
able to gracefully respond to such changes now.
Using chroot to enforce read-only file-system access
====================================================
By placing a chroot component in-between a file-system client and server, the
client's view on the file system can be limited to a specific directory. With
the current release, chroot can additionally be used to restrict a writeable
file-system session to become read-only. This is accomplished by the new
'writeable' attribute of chroot's policy nodes. By default, it is set to "no".
API changes
===========
Noncopyable AVL node/tree
-------------------------
Copying an AVL node generally violates the integrity of the corresponding
tree. To rule out subtle bugs where AVL nodes are accidentally copied, AVL
nodes are no longer copyable.
New 'Buffered_xml' utility
--------------------------
The 'Buffered_xml' utility located at _os/buffered_xml.h_ simplifies the
implementation of dynamically reconfigurable components that need to keep a
verbatim copy of certain parts of their configuration during configuration
updates.
New 'List_model' utility
------------------------
More and more components respond to dynamic configuration updates. For most
components, such updates are quite simple: replace an old internal state by a
new one. But in cases like init, menu_view, or window decorator, a
differential update is in order. Until now, each of these components employed
custom code for this task. As this code is not trivial, a common solution is
preferable. This solution comes in the form of the new 'List_model' utility
located at _base/include/util/list_model.h_. It introduces a light-weight
formalism to feed a component-internal data model from an externally-provided
XML structure.
Dynamically expandable reporter utility
---------------------------------------
In many cases, components that generate reports don't explicitly handle the
situation where the default buffer size of 4096 bytes is exceeded by the
report. This problem is easy to miss because reports are often small at
testing time but become larger when deployed in complex scenarios. In most
cases, the best way to handle an 'Xml_generator::Buffer_exceeded' exception is
upgrading the report session. The new 'Expanding_reporter' that accompanies
the original 'Reporter' in _os/reporter.h_ eases the handling of this common
case.
Languages and runtime environments
##################################
Nim programming language
========================
A new Nim library for constructing Genode servers is now available in the
World repository. This module provides utilities for the asynchronous
session-creation procedure introduced in the
[https://genode.org/documentation/release-notes/16.11#New_session-creation_procedure - 16.11]
release. Some introductory code snippets are provided here for the
adventurous.
An example of server creation using the 'genodeservers' module:
! import romclient, genodeservers
!
! var
! sessionsRom = newRomClient "session_requests"
! # synchronously open a ROM client to the parent
! romContent = sessionsRom.stream.readAll()
! # copy the ROM content to a heap string
! requestsParser = initSessionRequestsParser(romContent)
! # a state machine for parsing 'session_requests' XML
!
! for id, service, label in requestsParser.create:
! # the `create` iterator provider for the parser
! # hides the details of parsing the XML data
! discard txBufSize = requestsParser.argInt "tx_buf_size"
! # extract typed session arguments from the current parser state
! discard label.lastLabelElement()
! # label handling utilities are provided
! if service == "MyService":
! myCreateSessionProc(id, label)
!
This module streamlines the handling of session metadata, but the developer
must still provide hand-crafted wrappers over the C++ methods for managing
RPC objects and passing session capabilities to the parent. Most notoriously
a global pointer symbol, `genodeEnv`, is used to expose the component
environment object. In the future, this will be replaced by a typed object
passed from runtime to an application entry procedure.
! type MySessionCapability {.
! importcpp: "My_session::Session_capability",
! header: "my_session/capability.h".}
! # import a capability type
!
! type MyNativeSessionBase {.
! importcpp: "My_session::Session_rpc_object",
! header: "my_session/rpc_object.h".}
! # import C++ session RPC object
!
! type MyNativeSession = Constructible[MyNativeSessionBase]
! # apply the C++ Constructible template to defer calling
! # the object constructor
!
! proc construct(cppObj: MyNativeSession) {.
! importcpp: "#.construct(*genodeEnv)".}
! # call the C++ constructor, passing the global Genode::Env
!
! proc manage(cppObj: MyNativeSession): MySessionCapability {.
! importcpp: "genodeEnv->ep().manage(*#)".}
! # call a method from the gobal Env, dereferencing
! # thru the Constructible template
!
! type MyNimSessionObj = ref object
! cppImpl: MyNativeSession
! cap: MySessionCapability
! id: SessionId
! # C++ RPC objects are best kept in native
! # reference-counted Nim objects
!
! proc manage(obj: MyNimSessionObj) =
! obj.cppImpl.construct() # call our wrapped constructor
! GC_ref(obj)
! # manually increase the reference count on our session
! # object to prevent the component entrypoint from
! # referencing an RPC object that has been lost and
! # freed from the heap
! obj.cap = obj.cppImpl.manage() # store our capability
!
! proc myCreateSessionProc(id: SessionId): MyNimSessionObj =
! result = new MyNimSessionObj
! # create our object on the heap
! result.manage()
! # construct and manage our RPC object
! result.id = id
! # store the session id from our parent
Procedures for calling Nim code from an RPC object, dissolving
and destructing RPC objects, and managing the session lifetime
are exercises left to the reader.
Updated VirtualBox
==================
Our VirtualBox port got updated from version 5.1.22 to version 5.1.32 in order
to leverage the security updates and improved audio support. Additionally the
boot time of Linux guests got improved by adjusting our custom virtualization
back end.
Libraries and applications
##########################
New trace-logging component
===========================
The new trace-logger component can be used to easily gather, process, and
export different types of tracing data. Furthermore, it marks the next step
towards a user framework that makes access to Genode's manifold tracing
abilities
([https://genode.org/documentation/release-notes/13.08#Light-weight_event_tracing - 13.08],
[https://genode.org/documentation/release-notes/13.11#Improved_event_tracing - 13.11],
[https://genode.org/documentation/release-notes/15.08#Enhanced_tracing_facilities - 15.08])
intuitive and convenient.
The component can filter the available tracing subjects according to session
label policies and thread names. The processing of the tracing data can then
be configured for each selected subject individually, for groups of subjects,
or for all subjects together. The resulting data is exported as log output.
This is an example configuration of the trace logger, which shows the default
value for each attribute (except policy.thread and policy.label):
! <config verbose="no"
! session_ram="10M"
! session_arg_buffer="4K"
! session_parent_levels="0"
! period_sec="5"
! activity="no"
! affinity="no"
! default_policy="null"
! default_buffer="4K">
!
! <policy label="init -> timer" />
! <policy label_suffix=" -> ram_fs" />
! <policy label_prefix="init -> encryption -> "
! thread="worker"
! policy="null"
! buffer="4K" />
! </config>
The most important features so far when it comes to processing the traced
data are:
* Trace CPU activity and affinity ('activity' and 'affinity' attribute),
* Install individual policies for the creation of further tracing data
('policy' attributes) for instance, 'rpc_name' for a log of issued RPC calls),
* Dimensioning the subject-local trace buffers and the frequency of Trace Logger
data examination ('buffer' and 'period' attributes), and
* Configure the session to the Tracing server ('session' attributes).
A comprehensive documentation of the trace-logger component can be found in
_os/src/app/trace_logger/README_. An example of how to use the component is
given through the run script _os/run/trace_logger.run_.
New component for extracting archives
=====================================
The new 'extract' component located at _libports/src/app/extract_ extracts
the content of an arbitrary number of tar.xz archives according to its
configuration. It is used by the depot-download subsystem described in
Section [On-target package installation and deployment]. The component
is accompanied by the run script _libports/run/extract.run_ that illustrates
its use.
New signature-checking tool based on GnuPG
==========================================
The on-target installation of software packages requires a way to verify
cryptographic signatures of downloaded content within a Genode system.
The new 'verify' component located at _ports/src/app/verify_ facilitates the
code of GnuPG to verify detached OpenPGP signatures against public keys.
Since GnuPG depends on libgcrypt and libgpg-error, ports of those libraries
were added to the libports repository. The component comes with the run
script _ports/run/verify.run_ that demonstrates its usage.
Fetchurl component for downloading files
========================================
Fetchurl is a component for downloading files from the network, based
on the curl library. It used to reside in the genode-world repository.
Since it has become a mandatory part of Genode's on-target software
installation mechanism, we have moved it to the _libports_ repository now.
Besides this relocation, fetchurl received a welcome modernization. In
particular, the new version uses the modern socket-fs infrastructure of
the libc instead of relying on the deprecated libc_lwip plugin as a hard-wired
dependency.
New interactive FLIF image viewer
=================================
A simple image viewing application for the FLIF lossless image format was
written from scratch using the FLIF reference decoder library. The viewer can
be used to interactively view a directory of images and supports animation of
GIF-like FLIF files.
Ported 3rd-party software
=========================
With the current release, the following 3rd-party software becomes available
on Genode:
:[https://www.libarchive.org/ - libarchive]: is a library for uncompressing
and extracting various archive formats. It nicely wraps format-specific
libraries like zlib behind a unified and easy-to-use API. The port can
be found in the _libports_ repository.
:[https://lz4.github.io/lz4/ - lz4] and [https://tukaani.org/xz/ - liblzma]:
implement modern compression algorithms as supported by libarchive.
Thanks to Ben Larson for contributing the port of these libraries.
:[https://www.tcl.tk/ - Tcl]: is used as scripting language for various
Genode tools. With the new 'check_abi' tool described in Section
[Automated ABI consistency checks], the Tcl shell 'tclsh' has become
a dependency of the build system. Therefore, we made 'tclsh' available as
noux package. Note, however, that this port comprises solely the
functionality needed for simple scripting.
:[http://flif.info/ - FLIF]: is a library for the Free Lossless Image
Format. Thanks to Emery Hemingway for making it available in the
genode-world repository.
:[https://github.com/json-c/json-c/wiki - JSON-C]:
is a library for processing JSON-formatted data. Thanks to
Johannes Kliemann for contributing the port to the genode-world
repository.
:[https://www.nlnetlabs.nl/projects/ldns/ - Drill (ldns)]:
provides a utility for DNS testing. Thanks to Emery Hemingway for adding it
to the genode-world repository as a side activity of improving Genode's
network stack.
Updated packages for the Noux runtime environment
=================================================
The current release updates the following noux packages: less (version 487),
grep (version 3.1), coreutils (version 8.29), tar (version 1.30), findutils
(version 4.6), which (version 2.21), sed (version 4.4), and bash (version
4.4.18). Thanks to Hinnerk van Bruinehsen for this welcome contribution.
Device drivers
##############
Ethernet-driver for i.MX-based Wandboard
========================================
The current release contains a port of the Linux kernel driver for the
Ethernet card family originally produced by Freescale. We followed our
established approach to tailor an independent device-driver environment (DDE)
for the specific driver. To profit from synergies with the existing drivers of
the _dde_linux_ repository, we took the Linux kernel 4.4.3 as reference.
For now the current version is limited to support the Wandboard Quad as this
is the i.MX-based board that is nightly tested by our infrastructure. The
support of other boards using the same IP core is planned for future releases.
The driver can be found in _dde_linux/src/drivers/nic/fec_. To test the driver,
no further configuration is needed and you can have a look at one of the
automatic network tests, like _lwip.run_, as a reference.
Platforms
#########
Execution on bare hardware (base-hw)
====================================
Thanks to Johannes Schlatow from the TU Braunschweig, the support of the
Zynq-7000 boards by our base-hw kernel got extended. It is now possible to use
all CPU cores instead of only the primary one.
Updated Muen separation kernel
==============================
The Muen SK port has been updated to the latest development version 0.9. The
most notable features and improvements are the Crash Audit facility and support
for MirageOS/Solo5 subjects which may be executed alongside Genode/base-hw.
Thanks to this feature, the Muen project has reached a milestone by
self-hosting the [https://muen.sk] website on a Muen system. Currently, the
network driver is provided by a Linux subject but with some work it should be
possible to replace it with a Genode/base-hw nic_drv in the future.
Further details regarding Muen v0.9 can be found in the project's release
notes [https://groups.google.com/forum/#!topic/muen-dev/FPL9sc4yaBE].
Updated seL4 kernel
===================
Our remaining patches regarding UEFI framebuffer support got integrated into
the upstream codebase of the seL4 kernel. Hence, we updated our seL4 port to
the upstream version containing our patches.
Build system and tools
######################
Package management
==================
The package-management tools introduced last year have become a vital part
of Genode's workflow.
:Package management documentation:
[https://genode.org/documentation/developer-resources/package_management]
Prompted by the development of the on-target installation and deployment
mechanism featured in the current release, the tools received the following
refinements:
:Use of tar.xz as archive format: This change significantly reduces the size of
published depot content compared to the previously used tar.gz format.
:Subdirectories for archive versions:
In the original version of the depot layout, archives were named as
'<archive-name>-<version>'. Hence, the depot - in particular the download
location - had directories that grew in two dimensions. First, when new
archives were added. Second, when new versions of existing archives were
added (usually corresponding to Genode's release cycle). In the mid-term,
this would have resulted in a huge number of directory entries, e.g., in the
_src/_ subdirectory. To avoid this problem, the new version uses the scheme
'<archive-name>/<version>' instead. This way, at the _src/_ level, each
archive has one subdirectory (the number of subdirectories corresponds to
the number of archives). Inside the subdirectory, there is one entry per
version.
:Controlled rebuild of binary archives:
When calling the depot/create tool for a binary archive with 'FORCE=1', the
underlying source archives are re-extracted and the binary archive is
rebuilt. This is usually done after local changes in the source tree to
apply version updates to depot archives as needed. However, the implicit
rebuild is superfluous whenever the source-version remains the same. This is
particular inconvenient when re-creating pkg archives that refer to a large
number of src archives. Here, all binaries referenced by the pkg archive are
rebuilt each time. The new 'REBUILD' argument allows the user to skip
superfluous rebuilds in such situations. Normally, 'FORCE=1' implies
'REBUILD=1'. However, by explicitly specifying 'REBUILD=', existing binary
archives whose versions remain unchanged are kept instead of being rebuilt.
Offline validation of XML configurations
========================================
The _tool/run_ tool now automatically checks configurations against
target-specific XML schemes. Each component may define a configuration
scheme-file in its _target.mk_ file as follows:
! CONFIG_XSD = my_config.xsd
When the run tool checks the configuration of an instance of Genode's init
component, it additionally iterates through all start nodes of this
configuration. For each start node, it checks whether the according component
provides a configuration-scheme file and, if so, applies it to the
configuration inside the start node. This is done recursively. I.e., also the
child configurations of a sub-init of a sub-init ... of the top-level init
are covered this way.
Whenever the run tool detects an error in one of the checked configurations,
it stops and points out the location of the error. By now, there exist
configuration schemes for the init, the NIC router, and the trace logger
components. Our intention is that every component that interprets its
configuration will eventually be accompanied by such a scheme - not only to
validate actual configuration input but also to serve as documentation for
users of the component.
Automated ABI consistency checks
================================
In [https://genode.org/documentation/release-notes/17.02#Genode_Application_Binary_Interface - version 17.02],
we introduced a kernel-agnostic ABI, which ultimately paved the ground for
Genode's package management. For the time being, the ABI is not set in stone.
It is expected to evolve for some time until it hopefully approaches ABI
stability in the mid term. Whenever Genode's API changes, the ABI may be
affected. For example, symbol sizes may grow. Until now, side effects on the
ABI had to be curated manually. In practice, however, such side effects are
too easy to miss. Therefore, the current release adds a mandatory ABI checking
step to the build process. A new _tool/check_abi_ tool is invoked whenever a
shared object is built. It reports flaws in the ABI definition (such as
duplicated symbols) as well as inconsistencies between a shared object and its
ABI.

766
doc/release_notes-18-05.txt
View File

@@ -1,766 +0,0 @@
===============================================
Release notes for the Genode OS Framework 18.05
===============================================
Genode Labs
The driver behind the release 18.05 is the rapid evolution of the Sculpt
general-purpose OS. Following the initial version from February, which was
targeted at early adopters, the new Sculpt for The Curious (TC) introduces a
much more welcoming and empowering user experience (Section
[Sculpt for The Curious]).
It goes without saying that the interactive and dynamic nature of the Sculpt
scenario puts a lot more pressure on Genode's components compared to static
workloads. For example, Sculpt calls for the dynamic adjustment of user-level
network routing, the dynamic detection and management of partitions and file
systems, the support of USB storage devices from diverse vendors, and a way to
adapt the visual appearance to a great variety of screen resolutions. Most
improvements described below are our responses to these challenges.
That said, the release is not short of new features either. E.g., it features
the initial port of OpenJDK's HotSpot VM for executing Java programs on Genode
directly (Section [Java language support]), improves the support for the NXP
i.MX family of SoCs (Section [NXP i.MX SoC]), and enhances the VFS with new
plugins for copy-on-write and the auditing of file accesses
(Section [New VFS plugins]).
The release is complemented by the annual update of the Genode Foundations
book (PDF), which covers the fundamentals of the framework in great detail
(Section [New revision of the Genode Foundations book]).
Sculpt for The Curious
######################
With Sculpt for The Curious (TC), Genode 18.05 features the second revision of
the Sculpt general-purpose OS. Compared to the initial version for Early
Adopters (EA), it features a new interactive system-management component that
streamlines common tasks like the management of storage devices, or
configuring the network connectivity. The highlights of the new version of the
base system image are:
* Live-customization of almost all aspects of the system,
* The ability to install and run software in memory only,
* Hotplugging of USB storage devices,
* New support for NVMe storage devices in addition to SATA disks,
* Interactive network configuration including Wifi connectivity,
* Interactive management and inspection of storage devices and partitions,
* The option to host a complete and customized Sculpt installation on a
USB stick,
* Automated on-demand installation of software packages with visual feedback,
* Scalable fonts that are automatically adjusted to the screen resolution, and
* UEFI boot supported by default.
The base image is extensible by downloadable software packages that may
originate from different sources, safeguarded by cryptographic signatures.
It contains several example subsystems as a starting point:
* Basic GUI components like a window manager, a scalable backdrop, a
font server, and a simple software-rendering demo,
* A light-weight noux runtime for executing command-line-based software
such as GNU coreutils, bash, and vim.
* A package for downloading the installer and a suitable virtual-machine
configuration for Debian Linux,
* VirtualBox running Debian Linux,
* An example for running libretro-based games,
* A disposable VM that runs Firefox on TinyCore Linux, executed either in
VirtualBox or the light-weight Seoul virtual-machine monitor,
* A Qt5-based text editor.
Please refer to the updated
[https://genode.org/documentation/articles/sculpt-tc - Sculpt documentation]
to explore Sculpt TC.
The Sculpt version included with the current release requires the user to
build a boot image by hand. Following the steps described in the
documentation, this procedure takes a few minutes. We plan to provide
downloadable boot images a few weeks down the road once Sculpt TC received
intensive day-to-day testing by the early adopters. Your feedback is very
welcome!
New revision of the Genode Foundations book
###########################################
The "Genode Foundations" book received its annual revision, which reflects
the evolution of the framework over the past year. Specifically, the changes
since the last year's edition are:
: <div class="visualClear"><!-- --></div>
: <p>
: <div style="clear: both; float: left; margin-right:20px;">
: <a class="internal-link" href="https://genode.org">
: <img class="image-inline" src="http://genode.org/documentation/genode-foundations-title.png">
: </a>
: </div>
: </p>
* Changed boot-loader infrastructure on PC hardware
* Package management
* Structural changes of Genode's custom base-hw kernel
* API improvements: Unicode handling, support for XML-based data models,
timeout-handling API
: <div class="visualClear"><!-- --></div>
To examine the changes in detail, please refer to the book's
[https://github.com/nfeske/genode-manual/commits/master - revision history].
Storage infrastructure
######################
VFS library and plugin interface
================================
The VFS (Virtual-File-System) library was expanded to meet new requirements
for the Sculpt scenario. The traditional file-system medium for component
state and configuration sculpting is the *ram_fs* server, but with the
limitation that files stored in the server are ephemeral. Any changes to
the initial state are lost when a system is shut down or the *ram_fs* server
is restarted. Now that persistent storage is usually served by a VFS plugin
hosted by the VFS server, it was a natural progression to introduce a means
for indicating VFS changes with 'File_system' session notifications. To this
end the VFS server was amended to send session notifications, and notification
support was added to the Rump and FatFs VFS plugins, allowing Ext2 and FAT
file-systems to host dynamic component state and configuration information.
Using the VFS for serving font data produced from files stored in the VFS made
it practical to allow VFS plugins to introspect the file system. Plugins now
have the means to access arbitrary paths from the file-system root or they may
host and expose their own internal file systems.
While the core of the VFS library is small compared to contemporaries in other
operating systems, the moment came to promote the VFS from a static to a
shared library. Components that use the C runtime have always loaded the VFS
dynamically as a subsystem of _libc.lib.so_, but native components carried the
bulk of its implementation. The VFS library is now provided as a shared
library and is included with the front-end server in the _src/vfs_ depot
archive. This change affects components that have been rebuilt against the
shared library but do not have their ROM policies updated to allow access to
the _vfs.lib.so_ ROM.
New VFS plugins
===============
File-system introspection has made two additional plugins possible, the *audit*
and *cow* plugins.
The *audit* plugin logs VFS paths as they are accessed to a dedicated LOG
session. This is useful for finding the files required by third-party
components without relying on documentation or auditing source code.
The *cow* plugin emulates copy-on-write behavior by copying the contents of
files lying in a read-only path to a read-write path as they are opened. This
plugin is considered a proof-of-concept and under-performing, but opens a way
of experimenting with seeding user-managed file-systems from immutable
file-system archives.
Plugins of this kind are most appropriately instantiated in the VFS server
with policies to restrict the intended components into paths provided by the
plugins. This prevents a component from escaping the effect of the plugin. An
example of "auditing" a libc component follows:
! <start name="audit_fs">
! <binary name="vfs"/>
! <config>
! <vfs>
! <dir name="data"> <!-- source files -->
! <tar "data.tar"/>
! <ram/>
! </dir>
! <dir name="audit"> <!-- virtual path that captures /data -->
! <audit path="/data"/>
! </dir>
! </vfs>
! <!-- route into virtual audit path -->
! <policy label_suffix="audit" root="/audit" writeable="yes"/>
! </config>
! </start>
!
! <start name="app">
! <config>
! <libc stdout="/log" stderr="/log"/>
! <vfs>
! <log/>
! <fs label="audit"/>
! </vfs>
! </config>
! </start>
Improved disk-partition discovery and access
============================================
The 'part_blk' component, which parses the partition table on a block device
and provides access to each partition through a block session, was extended to
make it easier to implement a management component on top of it. It now
features additional attributes in its report. For one the block size of each
partition as well as the type of the file system on the partition are
reported. The file system probing implementation is minimal and only contains
file systems that are commonly used on Genode systems, i.e., FAT32 and Ext2.
Furthermore, on GPT formatted disks, each partition has an 'expandable'
attribute that contains the number of blocks by which the partition can be
grown. The following exemplary report illustrates the adjustments:
!<partitions type="gpt" total_blocks="500118192" gpt_total="500118125" gpt_used="302254080">
! <partition number="1" name="BIOS boot partition"
! type="21686148-6449-6e6f-744e-656564454649" guid="db0701aa-02ae-474d-92d0-82738bfce5d2"
! start="2048" length="2048" block_size="512"/>
! <partition number="2" name="EFI System"
! type="c12a7328-f81f-11d2-ba4b-00a0c93ec93b" guid="74e43226-2afb-4575-bdda-83bf72f5a6e7"
! start="4096" length="262144" block_size="512" file_system="FAT32"/>
! <partition number="3" name="GENODE"
! type="0fc63daf-8483-4772-8e79-3d69d8477de4" guid="a950091d-87ba-4800-85bf-7b6a58abe6d5"
! start="235147264" length="67108864" block_size="512" file_system="Ext2"
! expandable="197862064"/>
!</partitions>
The heuristics of how the component probes the partition table were also
loosened. Instead of explicitly enabling support for GPT, the component will
now always try to parse the MBR as well as the GPT. It will bail out if both
are considered valid since using GPT/MBR hybrid tables is not supported and it
should be up to the user to make an educated decision. In cases where there is
no partition table, a 'partitions' report of 'type="disk"' will be generated
in which the complete disk is presented as partition number '0'. This is
needed as compatibility fallback for Sculpt EA installations.
Creating and modifying GUID partition tables
============================================
Part of the enhancements of Sculpt TC is the ability to manipulate the block
device used by Sculpt. We implemented a component called 'gpt_write', which
can create and modify a GPT and its entries. It considers alignment
constraints to make better use of 512e devices. It will, however, not perform
any boundary checking. It does not handle overlapping partitions and only when
applying a partition, it makes sure that the partition will fit. The following
configuration illustrates its operation:
!<start name="gpt_write">
! <resource name="RAM" quantum="2M"/>
! <config verbose="yes" initialize="yes" align="4K">
! <actions>
! <add entry="1" type="BIOS" label="GRUB BIOS" start="2048" size="1M"/>
! <add entry="2" type="EFI" label="EFI System" start="4096" size="16M"/>
! <add entry="3" type="Linux" label="GENODE" start="36864" size="128M"/>
! <add type="BDP" label="FAT32 Data" size="max"/>
! <delete entry="1"/>
! <delete label="FAT32 Data"/>
! <modify label="GENODE" new_label="GENODE*" new_size="max"/>
! </actions>
! </config>
!</start>
Please read _repos/gems/src/app/gpt_write/README_ for more detailed information
on how to use the component and feel free to check out the run script
_repos/gems/run/gpt_write.run_.
User-level networking
#####################
NIC router
==========
The NIC router has received major improvements that were mainly motivated by
our daily experience with the Sculpt scenario where the router serves as NAPT
component in front of the virtual machines that host our work OS's. In this
role, it is subject to a permanent load driven by real-world tasks.
Furthermore, it has to have a user interface that makes it a pleasant
experience to deploy in a dynamic environment. This led to our primary goal:
We had to overcome the need to restart the NIC router, and thereby all
components that depend on it, whenever its configuration changes and while
doing so, not to interrupt the communication of its client unnecessarily.
We managed to make the NIC router fully re-configurable at runtime in a way
that it always tries to keep as much state information as possible throughout
the process. This means that network communication going through the NIC
router is not affected by a configuration update unless the configuration
change affects parts that were involved in an existing communication channel.
One prerequisite for this feature was that NIC session clients can connect at
any time to the NIC router regardless of whether there is a matching domain
for the session or not. As long as a session has no domain, the NIC router
does not send any packet to it and drops all packets coming from it. But, at
least, the session and the corresponding client component stay alive, even if
their already assigned domain disappears with a new configuration.
At the uplink, in contrast, the lifetime of the session remains bound to the
lifetime of the domain. The uplink domain-tag received a new attribute
named 'label' (only considered at the domain-tag of the uplink). It denotes
the label of the uplink session. With these two particularities of the uplink
domain, one can now easily switch between different NIC session servers. The
NIC router will close and request the corresponding NIC session with the
current 'label' value if the 'domain' node is removed/added or the label
changes. Thereby, the NIC router can now be used to dynamically switch between
network interfaces like wireless and wired adapters.
Furthermore, we improved the NIC router's ability to handle DNS server
information. Domains can wait for the DNS server info of the DHCP client of
another domain. This is done with the new attribute 'dns_server_from' in the
'<dhcp_server>' tag. Each time the DNS server info of the remote domain
changes, the DHCP server with the 'dns_server_from' attribute will toggle the
link state of each session at its domain. This can be used by clients as a
hint to request their DHCP info anew from the NIC router and thereby receive
the updated DNS server information.
When it comes to protocols, the most notable change is that the NIC router now
also supports routing and NAPT for ICMP. With the new '<icmp>' sub node of the
'<domain>' tag, ICMP routes to other domains can be created. Instead of ports,
the ICMP IDs are used for NAPT. Similar to the 'udp-ports' and 'tcp-ports'
attributes, the size of the ID space for each NAPT client is configured via
the new 'icmp-ids' attribute in the '<nat>' tag.
Last but not least, the following small features were also added to the NIC
router:
:Attribute 'verbose_packets' for the '<config>' and the '<domain>' node:
Toggles the logging of most important protocol header fields globally or
domain-locally. The 'verbose' attribute does not affect this kind of debug
output anymore.
:Report DNS server info:
If the 'config' attribute in the '<report>' node is enabled, the NIC router
will now also report the DNS server info for each domain.
:Attribute 'config_triggers' in the '<report>' node:
Toggles whether the NIC router immediately sends a report whenever the IPv4
configuration of a domain changes, regardless of any timeouts.
:IPv4 point-to-point support:
If a domain receives an IP configuration with a subnet mask of
255.255.255.255 it will switch to point-to-point IPv4 (requires a valid
gateway address at the domain).
:ICMP destination unreachable on non-routable packets:
The NIC router now responds with an ICMP "destination unreachable" packet to
packets that are not routable at an interface with a domain.
For more information, have a look at the _os/src/server/nic_router/README_
file. Examples can be found in the run scripts
_dde_linux/run/nic_router_uplinks.run_,
_libports/run/nic_router_dyn_config.run_, and _os/run/ping_nic_router.run_.
NIC dump
========
The output level of the NIC dump component can now be configured per protocol
by using the protocol names as attributes: 'eth', 'arp', 'ipv4', 'dhcp',
'udp', 'icmp', and 'tcp'.
The available debug levels are:
:no: Do not print out this protocol.
:name: Print only the protocol name.
:default: Print a short summary of the most important header values.
:all: Print all available header values.
Additionally, you can set a default debug level for protocols that are not
configured using the 'default' attribute.
For more information, please refer to _os/src/server/nic_dump/README_.
GUI stack
#########
With Sculpt becoming more and more end-user oriented, Genode's GUI stack came
into focus. It was time to reconsider several interim solutions that worked
well in the past but would not scale up to a modern general-purpose OS. Two
concrete examples are the support of scalable fonts and Unicode characters. In
the past, Genode used to restrict textual output to the Latin-1 character set
and employed pixel-based fonts only. The current release overcomes these
limitations by featuring completely new text-output facilities.
UTF-8 support and improved text rendering
=========================================
The UTF-8 text encoding overcomes the severely limited code-point range of the
ASCII and Latin-1 character sets by representing characters by a varying
number of bytes. Today, UTF-8 is generally considered as the standard encoding
for text. The new UTF-8 decoder at _os/util/utf8.h_ clears the path for
Genode's native GUI components to follow suit. The first beneficiary is
Genode's graphical terminal, which has become able to display Unicode
characters and pass user input as UTF-8-encoded data to its terminal-session
client.
Terminal enhancements
=====================
Speaking of the graphical terminal, the current incarnation got a welcome
overhaul. First, we reduced its complexity by removing obsolete features like
built-in keyboard-layout handling, which are no longer needed when combining
the terminal with our modern input-filter component. Furthermore, the terminal
has become dynamically resizeable, forwarding screen-size changes to the
terminal client. Should the client be a Noux runtime, such a change is
reflected to the running application as a SIG_WINCH signal. The application -
e.g., Vim - responds to the signal by requesting the new terminal size.
Finally, the terminal protocol was changed from 'linux' escape sequences to
'screen' escape sequences in the anticipation of making the terminal more
flexible in the future.
Text rendering
==============
Throughout Genode, many GUI components reused the text-output utilities
of the nitpicker GUI server. These utilities, however, relied on a simple
pixel font format. To make the text output more flexible, nitpicker's text
painter located at _nitpicker_gfx/text_painter.h_ has been replaced by a
completely new implementation that decouples the font format from the
glyph rendering and takes UTF-8 strings as input. In the process, the glyph
rendering got a lot more sophisticated, supporting horizontal sub-pixel
positioning and filtering.
Font-format support
===================
To remove the omnipresent use of fixed-size pixel fonts throughout Genode,
the following new components entered the picture:
First, the new 'ttf_font' library implements nitpicker's font interface by
using the TrueType renderer of the STB single-header library.
Second, the new 'vfs_ttf' VFS plugin uses the 'ttf_font' library to export a
rendered TrueType font as a virtual file system. The various font properties
as well as the actual glyph images become accessible as regular files. This
way, an application that needs to draw text can read the glyph data directly
from its VFS instead of depending on a font-rendering library.
Third, the new 'Vfs_font' utility located at _gems/include/gems/vfs_font.h_
implements nitpicker's font interface by obtaining the glyphs from the
component-local VFS. It is complemented by the 'Cached_font' utility, which
implements an LRU glyph cache.
With this infrastructure in place, several existing GUI components could
be updated, most prominently the graphical terminal and the menu-view
widget-rendering engine. By facilitating the VFS as interface for propagating
glyph data, components no longer need to manage fonts and their configuration
individually. They just access their VFS. When integrating the component into
a scenario, one can decide whether to mount a font-rendering library directly
at the component, or - alternatively - route a file-system session to a
central font server. The latter is just a regular VFS server with the fonts
mounted as pseudo file systems. Since the glyph renderer is a VFS plugin, it
could be replaced by another implementation in the future without touching any
component.
Modernized API for input-event processing
=========================================
Genode's input-session interface changed very little over the years. Even
though it received evolutionary enhancements from time to time, its design
resembled a traditional C-style interface from the medieval era. We found that
the interface left too much room for interpretation. In particular, the meta
data per event type was defined in a rather ad-hoc way, which raised
uncertainties. For example, is a button-press event accompanied with a
positional value or not? To remove these uncertainties, the current release
replaces the 'Input::Event', with a new implementation that facilitates a safe
way of accessing event meta data. Besides this design change, there is one
noteworthy semantic change as well. With the new interface, symbolic character
information are provided along with their corresponding press events rather
than as distinct events, which - according to our practical findings - greatly
simplifies the consumer side of the 'Input::Event' interface.
Improved keyboard-focus handling
================================
The nitpicker GUI server multiplexes one screen among multiple GUI clients in
a secure way. One aspect remained underdeveloped so far, which is the keyboard
focus handling. Nitpicker's 'Session:focus' call previously triggered a one-off
focus change at call time. This focus change did not pass the same code paths
as a focus change triggered by a "focus" ROM update, which led to
inconsistencies.
The new version changes the implementation of 'Session::focus' such that the
relationship of the caller and the focused session is preserved beyond the
call time. Whenever the calling session is focused in the future, the
specified session will receive the focus instead. So 'Session::focus' no
longer represents a single operation but propagates the information about the
inter-session relationship. This information is taken into account whenever
the focus is evaluated regardless of how the change is triggered. This makes
the focus handling in scenarios like the window manager more robust.
Device drivers
##############
NVMe storage devices
====================
Since NVMe devices have become common in contemporary systems, it is time to
provide a driver for such devices on Genode. With this release, we introduce a
component that is able to drive consumer-grade NVMe storage devices, i.e.,
there is no support for namespace management or other enterprise-grade
features. For now, to keep things simple, the driver uses the device in an
old-fashioned way and uses only one I/O queue with at most 128 entries. That
is to say it does not exploit the parallelism necessary to unlock the full
potential of NVMe storage. Nonetheless, it performs well. The following
snippet illustrates its configuration:
!<start name="nvme_drv">
! <resource name="ram" quantum="8M"/>
! <provides><service name="Block"/></provides>
! <config>
! <report namespace="yes"/>
! <policy label_prefix="client1" writeable="yes"/>
! </config>
!</start>
The component will generate a report, which contains all active namespaces, if
reporting is enabled by setting the 'namespace' attribute of the '<report>'
node to 'yes'. A report may look like the following example:
!<controller model="QEMU NVMe Ctrl" serial="FNRD">
! <namespace id="1" block_count="32768" block_size="512"/>
!</controller>
For an example on how to integrate this component, please have a look at the
_repos/os/run/nvme.run_ script.
While implementing the NVMe driver, a new component for testing block-sessions
was used. In contrast to the already existing 'blk_bench' and 'blk_cli'
components, it features a variety of different test patterns, which can be
selected in its configuration and can be used to test a block component more
thoroughly. For more information please refer to
_repos/os/src/app/block_tester/README_
NXP i.MX SoC
============
We extended the Linux kernel driver port for Ethernet cards found in NXP i.MX
SoC, which was introduced in the previous release. Now does it not only
support i.MX6Q SoC based boards like the Wandboard, but the i.MX53 and i.MX6SX
SoC as well. The new driver was successfully tested with the i.MX53 Quick
Start Board and the Nitrogen6 SOLOX. The latter board even contains two
Ethernet cards. But due to technical limitations of the board design, the same
driver instance has to be used for both cards. Currently, the driver is
tweaked to run on different boards via its configuration ROM. When no
configuration is provided, it appropriates the values for successfully
executing on the Wandboard. The following is an example configuration for the
i.MX53:
! <config>
! <card name="fec0" type="fsl,imx25-fec" mii="rmii" irq="87" mmio="0x63fec000"/>
! </config>
As a side effect of enabling networking on the Nitrogen6 SOLOX, support for
GPIO based signals has been added to the framework too. The existing GPIO
driver for i.MX53 SoC got extended to additionally support the i.MX6 family.
There are some known limitations when using different drivers like Ethernet
and SD-card drivers on the Wandboard right now. At the moment, those drivers
adjust clock parameters and I/O pin configurations independently from each
other, which can lead to inconsistencies. We plan to address those issues with
the implementation of a platform driver for the i.MX6 SoC family.
Improved USB-storage driver
===========================
We improved the stability of the USB-storage driver (usb_block_drv) and
made it compatible with a lot more devices as the driver has become a pivotal
ingredient of the Sculpt scenario. Due to the changes, the way the driver
operates has changed. On the one hand, now it first tries to use 10-byte
Command Descriptor Blocks (CDB) in its SCSI layer and will only switch to
16-byte CDBs when it encounters a device whose blocks cannot be completely
accessed via the former descriptor size. On the other hand, because some
tested devices stopped working after issuing a USB device reset, the reset was
made optional. By setting the 'reset_device' attribute in the '<config>' node
to 'yes', the driver is instructed to perform the USB device reset.
Libraries and applications
##########################
Packaged Qt5 framework
======================
We created package recipes for all previously ported Qt5 libraries and their
dependencies and adapted the run scripts accordingly. Please note that the
host tools needed for building Qt applications (moc, rcc, uic) are not built
automatically anymore, but need to be built and installed manually with the
new 'tool/tool_chain_qt5' script.
Java language support
=====================
Over the course of the past year, we started to look into Java support for
Genode with the ultimate goal of porting an existing Java Virtual Machine
(JVM), which translates and executes Java byte code, to Genode. After
investigating possible JVM candidates, it became obvious that
[http://openjdk.java.net - OpenJDK] is the only viable option when looking for
a functional, maintained, feature complete, and open-source Java SDK.
Therefore, we decided upon OpenJDK version 9 and started to port OpenJDK's
HotSpot virtual machine.
In the first step, we followed the approach to enable HotSpot's internal
Just-in Time (JIT) compiler, which translates byte code into machine code and
is the option with the most to offer performance wise. But we also wanted
support for ARM platforms and soon realized, there was almost no JIT compiler
support for ARM other than for Linux. The Linux version is deeply integrated
into the Linux system libraries (e.g., glibc), which makes it very hard to
bring the compiler onto Genode. For example, Genode uses FreeBSD's libc and
that would now have to offer glibc semantics.
After additional research, we found the so-called interpreter version of the
HotSpot VM. This version does not compile byte code, but interprets and
emulates the code at runtime. It is of course slower than the JIT compiler
version, but also machine-architecture independent, so the same HotSpot VM can
be compiled for x86 and ARM platforms. With the JVM running on Genode, we
added networking and file-system access support via Genode's VFS layer. Note,
there is no graphical toolkit support as of now, but most standard library
classes should work. Also, the byte code has to be compiled on a different
host system (e.g., Linux, *BSD) as of now, since we did not bring the Java
compiler to Genode.
To give Java a spin, a run script can be found under _ports/run/java.run_.
Ada language support
====================
Support for components and libraries written in the Ada/SPARK programming
language experienced a rework with the final goal of seamless integration with
the base framework. We added a new _ada_ library, which contains a (currently
minimal) runtime taken from the sources of our GCC port and thus is always
consistent with the tool chain in use. It is built as a shared library
_ada.lib.so_ that needs to be added to the list of boot modules.
The example in _libports/src/test/ada_ showcases the implementation of an Ada
component using a custom library _test-ada_, which is also implemented in Ada.
Seoul VMM on NOVA
=================
The Seoul/Vancouver VMM - introduced to Genode with release 11.11 - received
some renovations to be able to run recent Linux VMs. Namely the output of the
guest during early boot is now visible and the network models got revised.
Additionally, the Seoul VMM has been packaged and can be used in Sculpt.
Ported software
===============
The [https://dnsprivacy.org/wiki/display/DP/DNS+Privacy+Daemon+-+Stubby - Stubby]
DNS daemon has been ported to begin experimentations with DNS as a native
service. There is a tendency for DNS configuration frameworks to diverge
between operating systems and releases, an inconvenience that is magnified
when maintaining virtual machines. Name-server configuration via DHCP has been
the only constant, so hosting DNS natively and configuring virtual-machines
with the *nic_router* DHCP server presents itself as a viable solution to the
guest resolver quagmire. Expect DNS services in later Sculpt releases.
Platforms
#########
Accessing PCI via ECAM/MMCONF
=============================
The platform driver on x86 is trusted with guarding access to PCI
devices. Up to now, I/O ports have been used to configure the PCI subsystem.
On modern x86 architectures, PCI devices can be configured by using Memory
Mapped I/O (MMIO). This method was introduced with PCI Express and is called
Enhanced Configuration Access Mechanism (ECAM). For Each PCI device a separate
4 KiB MMIO page exists to serves as the configuration interface between OS and
PCI device.
The exact location of all the 4K MMIO pages of the PCI devices is machine
specific and must be determined during the bootstrap phase. The ACPI driver on
Genode is in charge of this procedure and reports the location of the
ECAM/MMCONF region to the platform driver via the 'acpi' ROM.
Besides using a modern PCI interface, switching to ECAM/MMCON served to ease
the execution of Genode/hw on top of the Muen separation kernel.
Kernel-agnostic platform-information handling
=============================================
Up to now, special kernel-specific information was propagated to components
such as Virtualbox, the Seoul VMM, and the timer by reusing the
kernel-provided data structures. For Genode/NOVA, the hypervisor info page
(HIP) was exported as an ordinary Genode ROM. With the rise of Sculpt and the
packaging of components in a - as far as possible - kernel-independent way,
the propagation of kernel-specific information became a stumbling block.
With this release we abandon the 'hypervisor_info_page' ROM of Genode/NOVA and
replace it with a Genode ROM called 'platform_info'. The 'platform_info' ROM
is planned to contain solely information about the host hardware, which may
not be gathered otherwise by Genode components. In the current state it
contains information required by VMMs, namely whether AMD SVM or Intel VMX is
available and usable. Additionally, the ROM contains information about the
frequency of the time stamp counter.
Updated seL4 kernel to version 9.0.1
====================================
Thanks to Hinnerk van Bruinehsen, the seL4 version used by Genode has been
updated to 9.0.1.
Updated Muen separation kernel
==============================
With the addition of memory-mapped access to the PCI config-space in Genode,
base-hw subjects on Muen now only see the effectively assigned physical
devices. This makes it possible to run Genode in parallel with other subjects
and to pass-through different PCI devices for each instance.
The Muen update also brings a much simplified subject info structure plus some
tweaks to the Muen system policy XML format to facilitate easier integration
of new hardware platform specifications.
Build system and tools
######################
Validating 3rd-party code downloads via SHA256
==============================================
This release removes support for verifying source code of third-party ports
with the SHA1 hash algorithm. Last year, SHA1 was banished as a credible
cryptographic hash function after the demonstration of a full collision
attack. Since the
[https://genode.org/documentation/release-notes/14.05 - 14.05 release],
port files have been verified using SHA1, this release replaces all file
digests with SHA256 digests. Any port definitions maintained in external
repositories are required to make these replacements as well. No collisions
have been discovered against source code archives but nonetheless there is an
obligation to widen our margin of safety.
Creating GPT-based disk images by default
=========================================
Up to now Genode's run tool was able to create x86 bootable images in three
flavours:
* Either as ISO bootable by BIOS legacy - 'image/iso', or as
* GPT partitioned disk image only bootable by UEFI - 'image/uefi', or as
* MBR partitioned disk image only bootable by BIOS legacy - 'image/disk'.
With Sculpt came the demand to have a single image type that is in principle
bootable by both UEFI and BIOS legacy. Additionally with Sculpt, we began to
prefer working with GPT partitioned devices.
In the light of the new demands, we changed the 'image/disk' run tool support
to create a GPT partitioned disk image bootable by a legacy BIOS and by UEFI.

1075
doc/release_notes-18-08.txt
View File

File diff suppressed because it is too large Load Diff

836
doc/release_notes-19-02.txt
View File

@@ -1,836 +0,0 @@
===============================================
Release notes for the Genode OS Framework 19.02
===============================================
Genode Labs
In our [https://genode.org/about/road-map - road map] for 2019, we stated
our goal to make Genode more relevant and appealing for a broader community.
The current release takes a big leap towards that goal: It opens up Sculpt
OS for 3rd-party software providers, introduces a federated blogging
platform about Genode-related topics, and makes the world's
[https://www.tiobe.com/tiobe-index/ - most popular] programming language
(Java) available to Genode users.
With the 4th stage of the evolution of Sculpt OS - themed as "community
experience" (CE) - Genode's custom general-purpose OS introduces a novel and
simple way for users to discover software, and for software providers to
announce software. There is no middle man like an app store or a distribution!
We hope that this federated model of software provisioning and deployment will
have a vitalizing effect on the community around Genode. On a practical level,
the interactivity of the new version is a playful and fun experience. Section
[Sculpt OS as a community experience (CE)] gives a rough overview about
*Sculpt CE*. A ready-to-use disk image will be released mid of March.
When speaking of "software providers", we don't think of anonymous
repositories. Software - and Free Software in particular - is developed and
provided by individuals after all. So in our federated way of software
distribution, we want to highlight this individuality. For this reason, we
launched a new blogging platform for Genode-related stories called
[https://genodians.org - Genodians.org], which gives everyone who is
enthusiastic about Genode - users and developers alike - a platform to express
ideas, announce software, or share practical tips and tricks. As explained in
the [https://genodians.org/nfeske/2019-01-07-welcome - initial posting],
*Genodians.org* is - in the spirit of Sculpt's software distribution model -
also organized in a federated way. The content is hosted and remains
completely under control by the respective authors. The website merely
aggregates and presents the content. It goes without saying that Genodians.org
is based on Genode. The system image that contains the entire web appliance -
from the kernel over the content management to the web server - is contained
in a 8 MiB disk image.
Section [Genodians.org as a showcase of a Genode-based web appliance] goes
into more detail.
We identified the support of popular programming languages as one key aspect
to attract a broader community. With the current release, our port of
*OpenJDK* (Section [Java]) for both 64-bit x86 and 32-bit ARM has reached a
mature state suitable for the creation of web services. This paves the way
towards quite exciting new system creations, like the *Boot2Java* system
presented in Section [Showcase of a Java-based network appliance].
On the account of programming-language support, we are happy to announce
a vastly improved runtime support for *Ada* and *SPARK* (Section
[Ada and SPARK] as well as the initial support for OCaml (Section [OCaml]).
Besides the work on shiny features, the current release cycle included a
profound *spring cleanup* described in Section
[Base framework and OS-level infrastructure].
It thereby finalizes the huge API modernization initiated almost three years
ago.
Sculpt OS as a community experience (CE)
########################################
When we laid out the road map for Sculpt OS a little more than one year ago,
we envisioned four stages of development. Sculpt EA was geared towards
die-hard early adopters facilitating the live editing of the system using a
text editor. Sculpt TC was targeted at "the curious" and included a graphical
user interface for common administrative tasks. Sculpt VC introduced the
visual composition of the system using an interactive graph. Even though
Sculpt already allowed for the installation of components from different
software providers at this stage, it offered no simple means to discover
software and the installation still required the manual editing of "launcher"
text files.
The final - community experience - stage of the plan ought to foster the
federated provisioning of software. Software providers should have the ability
to announce new packages. Conversely, users should be able to "subscribe" to
such announcements, similar how one would subscribe to an RSS feed. Once
software is discovered in this way, it should take only a few clicks to
install and integrate it into the running Sculpt system. No command-line
interface should stand in the way of discovery.
Sculpt CE is the realization of this idea.
For Sculpt users, the discovery starts at the '+' menu. In contrast to Sculpt
VC where the menu offered a rather long list of components to choose from, the
menu now offers nothing to be excited about.
[image sculpt_ce_menu]
However, at the very bottom, there is a new menu entry "Depot ...", which
leads to a sub menu with a list of enabled software providers. If connected to
the network, it also shows an entry named "Selection ...".
[image sculpt_ce_menu_depot]
The "Selection" sub menu allows the user to enable software providers.
[image sculpt_ce_selection]
When clicking on one of the checkboxes, the package index of the provider is
downloaded and the checkbox is highlighted. In the example, "nfeske" is
already enabled.
By clicking on the top-left triangle, one can always go one menu up. Back in
the depot menu, selecting a software provider will now display the index of
available packages. When selecting a new package, it can be installed via a
single click:
[image sculpt_ce_install]
Once the package is installed, the menu takes the user to a dialog for
integrating the package as a new component into the running Sculpt system. For
each resource required by the package, the user can decide how to connect it.
For example, the file system to be mounted at _/config/_ of a fresh
noux-system instance can be routed to any file system service the user wishes.
The noux-system won't know which file system is selected. It will just work
with it.
[image sculpt_ce_routes]
As soon as all routes are defined, the dialog presents a button for adding the
component to the system.
Using this interactive work flow, the discovery of software and its
integration becomes a rather playful process.
To make a once composed system permanent, one can use the inspect window to
copy the _/config/managed/deploy_ file to the _config/19.02/ directory of your
Genode partition. This way, the deployment configuration will take effect
immediately at boot time.
Genode 19.02 comes with Sculpt CE included. As usual, we will take a bit
of time following the release for thorough testing and refining before
announcing an updated disk image mid March. We would greatly appreciate
your feedback during this phase! For baking a fresh Sculpt image, please
refer to the
[https://genode.org/documentation/articles/sculpt-vc - documentation]
of Sculpt VC, but using the Git tag '19.02' instead of 'sculpt_vc'.
Announcing software packages
----------------------------
The community experience of Sculpt CE will ultimately depend on the
participation of software providers. To become listed in the selection dialog
mentioned above, you may consider including your public key and download
location at the Genode repository
[https://github.com/genodelabs/genode/tree/master/depot - at /depot/].
For the announcement of packages, a software provider can publish so-called
"index" files for a particular Sculpt version in the software provider's
depot. E.g., the index of packages supported on Sculpt 19.02 would be located
at _index/19.02_ within the depot. Like any other depot content, index files
are digitally signed. An example index for the software provider "nfeske"
would look as follows:
! <index>
! <index name="GUI">
! <pkg path="nfeske/pkg/sticks_blue_backdrop/2019-02-22"
! info="default desktop background"/>
! <pkg path="nfeske/pkg/themed_wm/2019-02-26"
! info="ready-to-use window manager"/>
! </index>
! <pkg path="nfeske/pkg/nano3d/2019-02-22"
! info="simple software-rendering demo"/>
! </index>
Each '<pkg>' node refers to a package with a concrete version and a short
description. By nesting '<index>' nodes, software categories can be defined.
Index files can be published like any other depot content using the
_depot/publish_ tool, which takes care about compressing and digitally signing
the published information:
| ./tool/depot/publish nfeske/index/19.02
To ease the updating of the index with current package versions, the
_sculpt.run_ script creates the _depot/index/<version>_ file from the input
found in _repos/gems/run/sculpt/index_. The latter file is void of any version
numbers.
For any questions about the process, please consult the Genode
[https://genode.org/community/mailing-lists - mailing list].
Showcase of a Java-based network appliance
##########################################
With OpenJDK in good shape (Section [Java]), we have created a Genode scenario
that demonstrates JVM's ability to execute well on embedded hardware. As
target platform, we choose an ARM (i.MX 6) based SoC with two integrated
network interface controllers. In the scenario, the Genode system boots
directly into a Java application, which in turn spawns two HTTP server
instances where each instance communicates through a dedicated NIC. Both
server instances run as one Java program.
[image java_nic_filter]
The Java application (a JAR file) is loaded from the board's SD card, and
therefore, can easily be replaced. If you are interested in Java and ARM SoC,
the full details and a step by step instruction can be found at our
[https://genodians.org/ssumpf/2019-02-27-java-19-02 - Genodians.org] site.
Genodians.org as a showcase of a Genode-based web appliance
###########################################################
[https://genodians.org - Genodians.org] is our take on a federated blogging
platform about Genode-related topics. It does not host the actual content
but rather aggregates content hosted elsewhere.
[image genodians]
The site periodically fetches content in the form of zip archives containing
raw text and PNG images, extracts the archives, and transforms the text into
HTML using a custom static site generator. The result of the transformation
process is served by the lighttpd web server.
This system is based on Genode and can be found here:
:Genodians.org repository:
[https://github.com/genodelabs/genodians.org]
A few noteworthy technical points about the site:
* The periodic process of downloading, extracting, and transforming content
is realized via the 'fetchurl', 'extract', and 'sequence' components.
* The textual content uses the markup of the
[https://github.com/nfeske/gosh - GOSH]
text processing tool, which is similar to
[https://daringfireball.net/projects/markdown/ - Markdown].
* The custom static site generator consists of a plain makefile and a few GOSH
style files. The makefile is executed within a noux instance, which is
Genode's custom Unix runtime environment. It is re-spawned for each
iteration. As GOSH is written in Tcl, the tclsh is used within the noux
environment.
* The lighttpd web server uses a statically supplied SSL certificate.
* The most time-consuming part of creating Genodians.org was the CSS
definition.
* The entire system image is about *8 MiB* in size. It includes the following
ingredients:
* Kernel (e.g., NOVA),
* Basic Genode components such as init, NIC router, the VFS server,
* Network driver (based on iPXE),
* Linux TCP/IP stack as a library,
* Curl, libssh, libssl, libcrypto, lighttpd
(for downloading and serving content),
* libarchive, zlib, liblzma (for extracting the downloaded content)
* Noux, coreutils (stripped down), bash, GNU make, and tclsh (for
transforming text into HTML)
More information about the site's inner working will be posted in a series
of articles - guess where? - at Genodians.org!
Base framework and OS-level infrastructure
##########################################
Removal of deprecated APIs
==========================
Almost three years ago -
with [https://genode.org/documentation/release-notes/16.05 - version 16.05] -
we started the transition to Genode's modern API. One year later - in
[https://genode.org/documentation/release-notes/17.05#Completed_component_transition_to_the_modern_API - version 17.05],
we announced the completion of this transition but we retained the deprecated
APIs to accommodate Genode users that picked up the new API only gradually.
With the current release, we finally drop the deprecated APIs along with a
couple of other legacies:
* The _base/timed_semaphore.h_ has been removed. In hindsight, officially
providing this utility was a big mistake because it lured developers into
a wrong direction. In fact, we found that there is no legitimate use of it
when a component is designed in a clean way. If a component relies on a
timed semaphore, it should better be redesigned. There are two noteworthy
places where a timed semaphore is still used as a band-aid solution:
the 'pthread_cond_wait' implementation of the libc, and DDE for rump
kernels. Those places now host a private copy of the timed semaphore, but
should ultimately be reworked.
* The header _base/printf.h_ has been removed along with the log back end
for 'printf'. The 'Console' with the format-string parser is still there
along with 'snprintf.h' because the latter is still used at a few places,
most prominently the 'Connection' classes.
* The notion of a 'Ram_session' exists no more since the former RAM-session
interface was merged into the PD-session interface in
[https://genode.org/documentation/release-notes/16.05#Consolidation_of_core_s_SIGNAL__CAP__RM__and_PD_services - version 16.05].
Still, the types were preserved (by typedefs to 'Pd_session') to keep up
API compatibility. Those last traces of 'Ram_session' are gone now. The
'Env::ram()' accessor returns the 'Ram_allocator' interface, which is a
subset of the 'Pd_session' interface.
* The use of the global 'Genode::env()' accessor function is not possible
anymore.
* The old 'Child_policy::resolve_session_request' interface that returned
a 'Service' instead of a 'Route' has been removed.
* Boolean accessor methods are no longer prefixed with 'is_'. E.g., instead
of 'is_valid()', use 'valid()'.
* All connection constructors need the 'Env' as argument.
* The 'Reporter' constructor needs an 'Env' argument now because the
reporter creates a report connection.
* The old notion of 'Signal_dispatcher' is gone. For receiving
asynchronous notifications, the 'Signal_handler' interface must be used.
* Transitional headers like _os/server.h_, _cap_session/_,
_volatile_object.h_, _os/attached*_dataspace.h_, and
_signal_rpc_dispatcher.h_ have been removed.
* The distinction between 'Thread_state' and 'Thread_state_base' does
not exist anymore. Only 'Thread_state' prevails.
* The header _cpu_thread/capability.h_ along with the type definition of
'Cpu_thread_capability' has been removed. Use the type
'Thread_capability' defined in cpu_session/cpu_session.h instead.
* The _os/ram_session_guard.h_ has been removed.
Use 'Constrained_ram_allocator' provided by _base/ram_allocator.h_ instead.
Source-tree reorganization
==========================
Timer moved from os to base repository
--------------------------------------
Traditionally, the user-level timer was hosted at the os repository at
_drivers/timer/_. However, since the timer and timeout handling have become
part of the base library (the dynamic linker), the component naturally belongs
to the base repository. It is now located at _base/src/timer/_ and
_base-<kernel>/src/timer/_ respectively.
Note that this change affects include paths for the former
_include/os/timer/_, _include/os/alarm.h_, and _include/os/duration.h_
headers. Those can now be found in _include/base/_.
Consistent naming of block components
-------------------------------------
Regarding the naming of files and APIs, Genode follows the convention of
avoiding abbreviations. Most components follow this convention, with the block
servers being the exception to the rule. Those were named "part_blk" or
"rom_blk". With the current release, we removed this inconsistency by renaming
those offenders, changing "blk" to "block". Closely related, abbreviations
like "cli" and "srv" have been replaced by "client" and "server".
Improved API safety
===================
XML-parsing API
---------------
Genode consistently uses XML for component configurations and for reports
generated by components. The latter are often consumed by other components.
This puts the XML parser into a prominent position. Genode's XML parser
comes in the form of the 'Xml_node' class. Since it was introduced before the
age of modern C++, it offers several risky "C-ish" methods, in particular
accessors that return pointers, raising memory-safety concerns. To promote a
safe programming style, the following parts of the interface are subject to
change now:
* The 'Xml_node::addr', 'Xml_node::content_addr', and 'Xml_node::content_base'
accessors will be removed because it is all too easy to store the returned
pointers and forget about the lifetime of the originating 'Xml_node' object.
Fortunately, in practice, those methods are rarely used because information
is typically represented in attributes, not as node content. However, to
still support the access to the raw content, the new
'Xml_node::with_raw_node', 'Xml_node::with_raw_content', and
'Xml_attribute::with_raw_value' methods call a functor taking the raw byte
buffer and size as arguments. This way, the lifetime of the pointer is
naturally bound to the scope of the functor.
* The new 'with_sub_node' method calls a functor with the specified
sub node as argument and thereby reduces the need for the traditional
'Xml_node::sub_node' method, which returns an 'Xml_node'.
The latter is risky because an 'Xml_node' contains a pointer to the actual
data. In contrast, the lifetime of the 'Xml_node' processed via the new
'Xml_node::with_sub_node' is naturally bound to the scope of the passed
functor.
* The 'Xml_attribute::value' and 'Xml_node::value' methods now take an
argument of type 'T &out' instead of 'T *out', which eliminates the
uncertainty of a possible nullptr argument.
The original interface will still be available for a while but it will
eventually be removed.
Simplified session-policy handling
----------------------------------
Most server components make use of the 'Session_policy' utility for selecting
a client policy depending on the session label. However, the pattern of its
use remained a bit inconsistent across components, in particular the handling
of the case where no policy could be found. Some components outright
denied the session where others implemented a fallback to a built-in default
policy. This inconsistency is now removed.
Following the principle of deny-by-default the absence of a matching policy
denies the session request. Since the 'Session_policy::No_policy_defined'
exception is a typedef to the 'Genode::Service_denied' exception, a server
does not need to explicitly handle it, which simplifies the implementation.
With this change, the 'No_policy_defined' case is always an error case. Hence,
the new version of 'Session_policy' prints a error message, which relieves the
server developer from implementing diagnostics in the server code.
As a consequence of this change, scenarios that used to rely on the
policy-fallback approach of some servers - notably the NIC bridge, window
manager, the window decorators - need a slight adaption: To enable the
fallback to a default policy, a '<default-policy>' node must be explicitly
specified in the server's configuration.
Removed pointers from Genode::Fifo interface
--------------------------------------------
To make the use of the 'Genode::Fifo' data structure more safe, all methods
that return pointers have been replaced by methods that call a functor with
a reference as argument.
New server-side block-request stream API
----------------------------------------
The current block-component API (_os/include/block/_) was designed at a time
long before Genode's modern component API was introduced. Back then, the use
of blocking calls was prevalent. Today, we design components to work
asynchronously. The original block-server API was successively enhanced to
allow the implementation of asynchronous block servers but it remained "upside
down" while growing more complicated than it should be.
To simplify the implementation and verification of block servers, the current
release introduces a modern API that will eventually replace the original
block-component API. The new API is called *block-request stream* and can be
found at
[https://github.com/genodelabs/genode/blob/master/repos/os/include/block/request_stream.h - os/include/block/request_stream.h].
It is designed with the following considerations:
* It anticipates the asynchronous operation of block servers by default.
Using the new API, such servers - in particular block-device drivers - can
be implemented as state machines triggered by client requests and device
interrupts.
* It reinforces the memory safety of the server code by not returning any
pointers or references.
* It relieves the server developers from handling special cases (like a
congested acknowledgement queue) while being flexible enough to accommodate
different categories of components like drivers, resource multiplexers
(part_block), and bump-in-the-wire components in a natural way.
* It naturally supports the batching of requests as well as zero-copy
(device DMA directly into the client's communication buffer).
The use of the new API is illustrated by the artificial test at
[https://github.com/genodelabs/genode/blob/master/repos/os/src/test/block_request_stream/main.cc - os/src/test/block_request_stream/].
We plan to successively migrate all existing block servers to this new API
and will remove the traditional block-component API eventually.
GUI stack
=========
Motivated by our work on Sculpt as described in Section
[Sculpt OS as a community experience (CE)], Genode's GUI stack received
the following improvements:
Window management
~~~~~~~~~~~~~~~~~
To improve the visual appearance of Sculpt's administrative GUI, the *themed*
*decorator* was enhanced with an option to disable decorations based on the
window label. By setting the 'decoration' attribute of a '<policy>' node to
"no", matching windows appear without any border, which is desirable for
Sculpt's component graph.
Furthermore, the themed decorator accepts the '<policy>' attribute
'motion=<number>'. The default value is 0. If a value higher than 0 is
specified, window-geometry changes are applied as an animation where the
<number> denotes the number of animation steps. This feature is used for
smoothing the placement of Sculpt's component graph.
The *motif* *decorator* - which is a nice alternative to the themed decorator -
is now giving visual feedback to mouse clicks. For example, while the user
drags a window by clicking on the window title, the title bar appears as
pressed, which creates an improved sense of responsiveness.
Unified shape-report routing
----------------------------
Applications propagate custom mouse-cursor shapes by issuing "shape" reports
towards the pointer component. The pointer component correlates the session
labels of the incoming shape reports with the label of nitpicker's currently
hovered client. Hence, an application's shape report session should take the
same route as its nitpicker session.
The presence of the window manager as an intermediary of the nitpicker session
breaks this rule. Consequently, rather awkward label-rewriting magic was
required when routing shape reports originating from windowed applications. To
make the shape reporting more natural, the window manager has become able to
proxy shape reports on behalf of its clients. When routing a shape through the
window manager, the labels of both the nitpicker sessions as well as the shape
report sessions contain the intermediary "wm ->" part.
Unicode support for the graphical terminal
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The terminal has been changed to represent characters as 16-bit codepoints
internally. It thereby became able to display a much larger variety of
international characters when using a suitable font. Unicodes are sent through
the terminal session via bursts of UTF-8 bytes now.
As a collateral change, the generic 'Codepoint' class became printable via
Genode's 'Output' interface. This way, a codepoint can easily be serialized
into UTF-8 bytes.
Programming languages
#####################
Ada and SPARK
=============
_The work described within this section was contributed by_
_[https://componolit.com - Componolit]. Thanks to Alexander Senier_
_and Johannes Kliemann for the fantastic collaboration!_
The integration of SPARK/Ada programs was improved greatly and the Ada runtime
was renamed to 'spark'. This emphasizes its main purpose: trusted components
that can be formally verified. A more feature-rich runtime based on 'libc' was
added to genode-world under the name 'ada'.
Build system integration
------------------------
Up until now, the (deprecated) 'gnatmake' tool was used to build Ada object
files. This was unfavorable for two reasons: First, multiple invocations of
gnatmake would sporadically corrupt the compiler-generated Ada linker files
('.ali') and break parallel builds. Second, Ada dependency information did not
get propagated into Genode's build system such that certain source changes
failed to trigger a rebuild.
For these reasons the 'gnatmake' tool has been dropped in favor of regular
compiler calls as done for all other languages. To facilitate consistent
rebuilds on source-code changes, the
[https://github.com/Componolit/ali2dep - ali2dep] tool was created. From an
'.ali' file, it produces '.d' files suitable for direct inclusion into
Genode's build system.
Until the integration into the toolchain, 'ali2dep' support needs to be
enabled through the 'CUSTOM_ALI2DEP' variable (absolute path or command name
if the command is in 'PATH'). By default, a warning about the absence of the
tool is emitted and dependency information is not generated. You can add this
variable to the 'etc/tools.conf' of your build directory as follows:
! CUSTOM_ALI2DEP = /path/to/ali2dep
Elaboration code
----------------
Previously, Ada programs that required elaboration code to be run were
unsupported on Genode. With this release, Ada programs are bound using
'gnatbind', which results in elaboration code being generated. An additional
benefit comes in the form of proper error messages at compile time if, for
example, source code is missing or outdated.
To run an Ada main program with elaboration, calls to 'adainit()' and
'adafinal()', generated by the binder, need to be added to your
component-construction code:
! extern "C" void _ada_main(void);
! extern "C" void adainit();
! extern "C" void adafinal();
!
! void Component::construct(Genode::Env &env)
! {
! adainit();
! _ada_main();
! adafinal();
! env.parent().exit(0);
! }
Note, that the name of the Ada main program ('_ada_main()') in this example
depends on the name of your main procedure.
Debug output
------------
Support for 'GNAT.IO' was added to the runtime. 'GNAT.IO' is a stripped-down
text I/O facility, which we map to a terminal session on Genode. Only output
is supported at the moment. A pointer to a terminal session has to be provided
to the ADA runtime in order to use 'GNAT.IO'. Please refer to
_repos/libports/src/test/gnatio/_ as an example.
Unit testing
------------
To facilitate test-driven development of Ada components, the
[https://www.adacore.com/documentation/aunit-cookbook - AUnit] unit testing
framework was ported to Genode. It is located in the genode-world repository
whereas a usage example can be found at 'src/test/aunit'. Note that the full
Ada runtime ('ada') is required as the framework uses features not present in
the SPARK runtime.
Java
====
Since Genode release
[https://genode.org/documentation/release-notes/18.11#Java_language_runtime - 18.11],
we continued our effort to enable the just-in-time (JIT) compiler for OpenJDK.
We are happy to announce that we were able to achieve this goal. The JIT
compiler is now enabled as default for both x86 (64 bit) and ARM (32 bit),
which significantly improves the performance of the Java virtual machine for
these architectures.
Additionally, we even further improved JVM's performance by taking advantage
of more aggressive compiler optimizations. This triggered some unidentified
bugs and led to a greatly enhanced stability of OpenJDK.
For a small hello world example we offer a simple run script:
! make run/java
For a more complex scenario please refer to Section
[Showcase of a Java-based network appliance].
Nim
===
Support for Nim has been removed from the Genode build system to encourage
building Nim components out-of-tree using the Nimble package manager.
The compatibility of the base system with the Nim runtime will be monitored
and improved regardless of this change.
OCaml
=====
A proof-of-concept port of the OCaml bytecode interpreter has been placed
in the world repository. This allows simple programs that are compiled to the
bytecode instructions to be executed and is a first step in supporting the
OCaml language. This interpreter does not yet include the standard library.
Therefore only language primitives are available. Porting the standard library
appears to simply be a matter of defining a build process. The greatest
hurdle to porting existing OCaml applications would seem to be finding a path
into the workflow of the OPAM package tooling.
Libraries and applications
##########################
New utility for taking screenshots
==================================
The flif_caputure screenshot utility has been added to the world repository.
This utility implements a framebuffer and input service which it proxies to
its parent. When the utility observes the *PrtSc* key, it captures the content
of the framebuffer and writes it to the file system in the FLIF image format,
FLIF being chosen primarily for its uncomplicated API. To use the utility in
practice, the common case would be to run it as a client of the default window
manager with a second window manager stack running as a client of the capture
utility. This allows capture behavior and scope to remain simple and explicit.
The background story behind the tool is covered by a dedicated
[https://genodians.org/ehmry/2019-01-31-flif_capture - posting] at Genodians.org.
Growing use of the Genode-World repository
==========================================
The [https://github.com/genodelabs/genode-world - Genode-World] repository
contains software that is not strictly part of the official Genode OS
framework but rather supplemental. In particular, it contains ports of
3rd-party software to Genode. The pool of ported software is steadily growing,
which moves the world repository more and more into the spotlight of Genode
users. For example, among a variety of games and experimental components, our
port of [Java - OpenJDK] is also hosted there. To acknowledge the growing
importance of the world repository, we added the building of all world depot
archives to our nightly build tests.
With this baseline of quality assurance in place, it was a good time to move
supplemental software such as Dosbox, FUSE, libav, libSDL, and its companion
libraries from the Genode repository to the Genode-World repository. Speaking
of libSDL, as part of the curation of the world content, the library back end
of libSDL has been changed to the direct use of the nitpicker session
interface instead of the lower-level framebuffer and input interfaces. Thanks
to this change, many scenarios could be greatly simplified, in particular
packages designated for the use in Sculpt OS.
Updated or removed 3rd-party software
=====================================
The following 3rd-party software received an update:
* OpenSSL updated to version 1.0.2q with 'SSL_CONF_*' enabled,
as needed by lighttpd's mod_openssl.
* Lighttpd updated to version 1.4.52, with TLS enabled.
* libpng updated to version 1.6.36
* jbig2dec updated to version 0.15
Removal of VirtualBox 4
-----------------------
All regular users of Genode's VirtualBox port migrated to version 5 a long
time ago. Version 4 was still maintained because this is the only version
supported on the Muen separation kernel. However, since the focus of the Muen
developers recently moved towards the use of nested virtualization, our
version of VirtualBox 4 is no longer vital for Muen. So we could remove it.
Platforms
#########
Board support for i.MX6 Quad Sabrelite and Nitrogen6 SoloX
==========================================================
In 2018, we extended our device-driver support for NXP i.MX 5 and 6 based ARM
platforms. This year, we continue our commitment to this platform with
additional support for the NXP reference board i.MX6 Quad Sabrelite and the
Nitrogen6 SoloX, which features dual Gigabit Ethernet.
Resizeable virtual framebuffer on Linux
=======================================
When executing Genode on Linux, we use a libSDL-based pseudo driver for
running graphical scenarios. This *fb_sdl* server has now been enhanced with
the ability to resize the SDL window. Such a resize event is translated into
Genode's framebuffer resize protocol. Thereby, the resizing of the host window
looks like a mode change to Genode's GUI stack. This greatly eases the testing
of the mode-change handling of components like the nitpicker GUI server.
Tooling and build system
########################
Enforced 'override' annotations
-------------------------------
The '-Wsuggest-override' warning complains about implementations of virtual
functions that lack the override keyword. If the implementation of a virtual
method is marked as 'override' the compiler checks for a matching virtual
method in the base class. If there is no such method, the implementation
unexpectedly diverged from the interface.
An interface may change over time, which is especially troublesome when it
contains a default implementation of a virtual method. Without 'override'
annotations, the compiler will silently add the outdated implementation of the
derived class as an overload of the interface's default implementation, which
introduces a subtle but potentially serious bug.
To rule out such bugs in the future, we made 'override' annotations mandatory
when using the default strict warning level.
Generalized use of the depot by run scripts
-------------------------------------------
With more and more run scripts using archives out of Genode's depot, we gained
experience with typical usage patterns. In particular, we found the universal
use of the configurable depot user preferable over the hard-wiring of
'genodelabs'. Hence the 'depot_user' function has become a built-in function
of the run tool.
Repeat mode for the depot-autopilot test environment
----------------------------------------------------
The _depot_autopilot.run_ script now supports the environment variable
'TEST_REPEAT' with the possible values 'until_forever' and 'until_failed'.
When specified, the tests are run in a loop. The latter argument is
particularly useful for reproducing sporadic errors.
New run script to execute a single test w/o the depot
-----------------------------------------------------
The new _os/run/test.run_ script allows for the quick execution of an
individual test from the build directory while side-stepping the depot.
It expects a 'PKG' variable specifying the test package. E.g.,
! make run/test KERNEL=nova PKG=test-xml_node
Note that it does not cover all test packages right now.
The tool and its limitations are explained in more detail by a dedicated
[https://genodians.org/nfeske/2019-02-05-shortcut-for-testing - posting]
at Genodians.org.
Support for undefined-behavior sanitizer
----------------------------------------
The
[https://developers.redhat.com/blog/2014/10/16/gcc-undefined-behavior-sanitizer-ubsan/ - UndefinedBehaviorSanitizer]
(UBSan), also described in the
[https://gcc.gnu.org/onlinedocs/gcc/Instrumentation-Options.html - GCC documentation],
can detect undefined behavior while a program is running.
With the current release, this diagnostic feature of GCC becomes available for
analyzing Genode components. By specifying 'SANITIZE_UNDEFINED = yes' in a
'target.mk' file, the '-fsanitize=undefined' compiler flag is enabled and the
program is linked with libubsan and libsanitizer_common. The program has to
call 'env.exec_static_constructors()' and 'sanitizer_init(env)' upon startup
to initialize the sanitizer libraries. Whenever undefined behavior is detected
while the program is running, a "runtime error:" message including the source
code location of the error is printed to the log.

830
doc/release_notes/08-11.txt Normal file
View File

@@ -0,0 +1,830 @@
==============================================
Release notes for the Genode OS Framework 8.11
==============================================
Genode Labs
Summary
#######
This document presents the new features and major changes introduced
in version 8.11 of the Genode OS Framework. It is geared towards
people interested in closely following the progress of the Genode
project and to developers who want to adopt their software to our
mainline development. The document aggregates important fragments
of the updated documentation such that you won't need to scan existing
documents for the new bits. Furthermore, it attempts to provide our
rationale behind the taken design decisions.
The general theme for the release 8.11 is enabling the use of the
Genode OS framework for real-world applications. Because we regard
the presence of device drivers and a way to reuse existing library
code as fundamental prerequisites for achieving this goal, the major
new additions are an API for device drivers written in C, an API for
handling asynchronous notifications, and a C runtime. Other noteworthy
improvements are the typification of capabilities at the C++-language
level, a way for receiving and handling application faults, the
introduction of managed dataspaces, and a new API for scheduling
timed events.
Base framework
##############
This section documents the new features and changes affecting the
'base' repository, in particular the base API.
New features
============
Connection handling
~~~~~~~~~~~~~~~~~~~
The interaction of a client with a server involves the definition of
session-construction arguments, the request of the session creation via
its parent, the initialization of the matching RPC-client stub code
with the received session capability, the actual use of the session
interface, and the closure of the session. A typical procedure of
using a service looks like this:
!#include <rom_session/client.h>
!...
!
!/* construct session-argument string and create session */
!char *args = "filename=config, ram_quota=4K");
!Capability session_cap = env()->parent()->session("ROM", args);
!
!/* initialize RPC stub code */
!Rom_session_client rsc(session_cap);
!
!/* invoke remote procedures, 'dataspace' is a RPC function */
!Capability ds_csp = rsc.dataspace();
!...
!
!/* call parent to close the session */
!env()->parent()->close(session_cap);
Even though this procedure does not seem to be overly complicated,
is has raised the following questions and criticism:
* The quota-donation argument is specific for each server. Most services
use client-donated RAM quota only for holding little meta data and,
thus, are happy with a donation of 4KB. Other services maintain larger
client-specific state and require higher RAM-quota donations. The
developer of a client has to be aware about the quota requirements for
each service used by his application.
* There exists no formalism for documenting session arguments.
* Because session arguments are passed to the 'session'-call as a plain
string, there are no syntax checks for the assembled string performed
at compile time. For example, a missing comma would go undetected until
a runtime test is performed.
* There are multiple lines of client code needed to open a session to
a service and the session capability must be maintained manually for
closing the session later on.
The new 'Connection' template provides a way to greatly simplify the
handling of session arguments, session creation, and destruction on the
client side. By implementing a service-specific connection class
inherited from 'Connection', session arguments become plain constructor
arguments, session functions can be called directly on the 'Connection'
object, and the session gets properly closed when destructing the
'Connection'. By convention, the 'Connection' class corresponding to a
service resides in a file called 'connection.h' in the directory of the
service's RPC interface. For each service, a corresponding 'Connection'
class becomes the natural place where session arguments and quota
donations are documented. With this new mechanism in place, the example
above becomes as simple as:
!#include <rom_session/connection.h>
!...
!
!/* create connection to the ROM service */
!Rom_connection rom("config");
!
!/* invoke remote procedure */
!Capability ds_csp = rom.dataspace();
[https://genode.org/documentation/api/base_index#Connecting_to_services - See the API documentation for the connection template...]
Typed capabilities
~~~~~~~~~~~~~~~~~~
A plain 'Capability' is an untyped reference to a remote object of any
type. For example, a capability can reference a thread object or a
session to a service. It is loosely similar to a C void pointer, for which
the programmer maintains the knowledge about which data type is actually
referenced. To facilitate the type-safe use of RPC interfaces at the C++
language level, we introduced a template for creating specialized
capability types ('Typed_capability' in 'base/typed_capability.h') and
the convention that each RPC interface declares a dedicated capability
type. Note that type-safety is not maintained across RPC interfaces. As
illustrated in Figure [layered_ipc], typification is done at the
object-framework level on the server side and via in the 'Connection'
classes at the client side.
[image layered_ipc]
From the application-developer's perspective, working with capabilities
has now become type-safe, making the produced code more readable and robust.
[https://genode.org/documentation/api/base_index#Capability_representation - See the updated API documentation for the capability representation...]
Fifo data structure
~~~~~~~~~~~~~~~~~~~
Because the 'List' data type inserts new list elements at the list head,
it cannot be used for implementing wait queues requiring first-in
first-out semantics. For such use cases, we introduced a dedicated
'Fifo' template. The main motivation for introducing 'Fifo' into the
base API is the new semaphore described below.
[https://genode.org/documentation/api/base_index#Structured_data_types - See the new API documentation for the fifo template...]
Semaphore
~~~~~~~~~
Alongside lock-based mutual exclusion of entering critical sections,
organizing threads in a producer-consumer relationship via a semaphore
is a common design pattern for thread synchronization. Prior versions
of Genode provided a preliminary semaphore implementation as part of
the 'os' repository. This implementation, however, supported only one
consumer thread (caller of the semaphore's 'down' function). We have
now enhanced our implementation to support multiple consumer threads
and added the semaphore to Genode's official base API. We have made
the wake-up policy in the presence of multiple consumers configurable
via a template argument. The default policy is first-in-first-out.
[https://genode.org/documentation/api/base_index#Synchronization - See the new API documentation for the semaphore...]
Thanks to Christian Prochaska for his valuable contributions to the new
semaphore design.
Asynchronous notifications
~~~~~~~~~~~~~~~~~~~~~~~~~~
Inter-process communication via remote procedure calls requires both
communication partners to operate in a synchronous fashion. The caller
of an RPC blocks as long as the RPC is not answered by the called
server. In order to receive the call, the server has to explicitly
wait for incoming messages. There are a number of situations where
synchronous communication is not suited.
For example, a GUI server wants to deliver a notification to one of its
clients about new input events being available. It does not want to
block on a RPC to one specific client because it has work to do for
other clients. Instead, the GUI server wants to deliver this
_notification_ with _fire-and-forget_ semantics and continue with
its operation immediately, regardless of whether the client received
the notification or not. The client, in turn, does not want to poll
for new input events at the GUI server but it wants to be _waken_up_
when something interesting happens. Another example is a block-device
driver that accepts many requests for read/write operations at once.
The operations may be processed out of order and may take a long time.
When having only synchronous communication available, the client and
the block device driver would have to employ one distinct thread for
each request, which is complicated and a waste of resources. Instead,
the block device driver just wants to acknowledge the completeness of
an operation _asynchronously_.
Because there are many more use cases for asynchronous inter-process
communication, we introduced a new signalling framework that complements
the existing synchronous RPC mode of communication with an interface for
issuing and receiving asynchronous notifications. It defines interfaces
for signal transmitters and signal receivers. A signal receiver can
receive signals from multiple sources, whereas the sources of incoming
signals are clearly distinguishable. One or multiple threads can either
poll or block for incoming signals. Each signal receiver is addressable
via a capability. The signal transmitter provides fire-and-forget
semantics for submitting signals to exactly one signal receiver. Signals
are communicated in a reliable fashion, which means that the exact number
of signals submitted to a signal transmitter is communicated to the
corresponding signal receiver. If notifications are generated at a higher
rate than as they can be processed at the receiver, the transmitter
counts the notifications and delivers the total amount with the next
signal transmission. This way, the total number of notifications gets
properly communicated to the receiver even if the receiver is not highly
responsive. Notifications do not carry any payload because this payload
would have to be queued at the transmitter.
[image signals]
Image [signals] illustrates the roles of signaller thread,
transmitter, receiver, and signal-handler thread.
[https://genode.org/documentation/api/base_index#Asynchronous_notifications - See the new API documentation for asynchronous notifications...]
The current generic implementation of the signalling API employs one
thread at each transmitter and one thread at each receiver. Because
the used threads are pretty heavy weight with regard to resource usage,
ports of Genode should replace this implementation with platform-
specific variants, for example by using inter-process semaphores or
native kernel support for signals.
Region-manager faults
~~~~~~~~~~~~~~~~~~~~~
In Genode, region-manager (RM) sessions are used to manage the
address-space layout for processes. A RM session is an address-space
layout that can be populated by attaching (portions of) dataspaces to
(regions of) the RM session. Normally, the RM session of a process is
first configured by the parent when decoding the process' ELF binary.
During the lifetime of the process, the process itself may attach
further dataspaces to its RM session to access the dataspace's content.
Core as the provider of the RM service uses this information for
resolving page faults raised by the process. In prior versions of
Genode, core ignored unresolvable page faults, printed a debug message
and halted the page-faulted thread. However, this condition may be of
interest, in particular to the process' parent for reacting on the
condition of a crashed child process. Therefore, we enhanced the RM
interface by a fault-handling mechanism. For each RM session, a fault
handler can be installed by registering a signal receiver capability.
If an unresolvable page fault occurs, core delivers a signal to the
registered fault handler. The fault handler, in turn, can request the
actual state of the RM session (page-fault address) and react upon
the fault. One possible reaction is attaching a new dataspace at the
fault address and thereby implicitly resolving the fault. If core
detects that a fault is resolved this way, it resumes the operation
of the faulted thread.
This mechanism works analogously to how page faults are handled by
CPUs, but on a more abstract level. A (n-level) page table corresponds
to a RM session, a page-table entry corresponds to a dataspace-
attachment, the RM-fault handler corresponds to a page-fault
exception handler, and the resolution of page-faults (RM fault)
follows the same basic scheme:
# Application accesses memory address with no valid page-table-entry
(RM fault)
# CPU generates page-fault exception (core delivers signal to fault
handler)
# Kernel reads exception-stack frame or special register to determine
fault address (RM-fault handler reads RM state)
# Kernel adds a valid page-table entry and returns from exception
(RM-fault handler attaches dataspace to RM session, core resumes
faulted thread)
The RM-fault mechanism is not only useful for detecting crashing child
processes but it enables a straight-forward implementation of growing
stacks and heap transparently for a child process. An example for
using RM-faults is provided at 'base/src/test/rm_fault'.
Note that this mechanism is only available on platforms on which core
resolves page faults. This is the case for kernels of the L4 family.
On Linux however, the Linux kernel resolves page faults and suspends
processes performing unresolvable memory accesses (segmentation fault).
Managed dataspaces (experimental)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The RM-fault mechanism clears the way for an exciting new feature
of Genode 8.11 called managed dataspaces. In prior versions of Genode,
each dataspace referred to a contiguous area of physical memory (or
memory-mapped I/O) obtained by one of core's RAM, ROM, or IO_MEM
services, hence we call them physical dataspaces. We have now added
a second type of dataspaces called managed dataspaces. In contrast
to a physical dataspace, a managed dataspace is backed by the content
described by an RM session. In fact, each RM session can be used as
dataspace and can thereby be attached to other RM sessions.
Combined with the RM fault mechanism described above, managed
dataspaces enable a new realm of applications such as dataspaces
entirely managed by user-level services, copy-on-write dataspaces,
non-contiguous large memory dataspaces that are immune to physical
memory fragmentation, process-local RM fault handlers (e.g., managing
the own thread-stack area as a sub-RM-session), and sparsely populated
dataspaces.
Current limitations
-------------------
Currently, managed dataspaces still have two major limitations. First,
this mechanism allows for creating cycles of RM sessions. Core must
detect such cycles during page-fault resolution. Although, a design for
an appropriate algorithm exists, cycle-detection is not yet implemented.
The missing cycle detection would enable a malicious process to force
core into an infinite loop. Second, RM faults are implemented using the
new signalling framework. With the current generic implementation, RM
sessions are far more resource-demanding than they should be. Once the
signalling framework is optimized for L4, RM sessions and thereby
managed dataspaces will become cheap. Until then, we do not recommend
to put this mechanism to heavy use.
Because of these current limitations, managed dataspaces are marked as
an experimental feature. When building Genode, experimental features are
disabled by default. To enable them, add a file called 'specs.conf'
with the following content to the 'etc/' subdirectory of your build
directory:
! SPECS += experimental
For an example of how to use the new mechanism to manage a part of a
process' own address space by itself, you may take a look at
'base/src/test/rm_nested'.
Changes
=======
Besides the addition of the new features described above, the following
parts of the base framework underwent changes worth describing.
Consistent use of typed capabilities and connection classes
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
We applied capability typification to all interfaces of Genode including
the base API and the interfaces defined in the 'os' repository. Figure
[base_cap_types] provides an overview about the capability types
provided by the base API.
[image base_cap_types]
Overview about the capability types provided by the base API
Furthermore, we have complemented all session interfaces with
appropriate 'Connection' classes taking service-specific session
arguments into account.
For session-interface classes, we introduced the convention to declare
the service name as part of the session-interface via a static member
function:
! static const char *service_name();
Allocator refinements
~~~~~~~~~~~~~~~~~~~~~
Throughout Genode, allocators are not only used for allocating memory
but also for managing address-space layouts and ranges of physical
resources such as I/O-port ranges or IRQ ranges. In these cases, the
address '0' may be a valid value. Consequently, this value cannot be
used to signal allocation errors as done in prior versions of Genode.
Furthermore, because managed dataspaces use the RM session interface to
define the dataspace layout, the address-'0' problem applies here as
well. We have now refined our allocator interfaces and the RM-session
interface to make them fit better for problems other than managing
virtual memory.
Misc changes
~~~~~~~~~~~~
We revised all interfaces to consistently use _exceptions_ to signal
error conditions rather than delivering error codes as return values.
This way, error codes become exception types that have a meaningful
name and, in contrast to global 'errno' definitions, an error exception
type can be defined local to the interface it applies to. Furthermore,
the use of exceptions allows for creating much cleaner looking interfaces.
Traditionally, we have provided our custom _printf_ implementation as C
symbol to make this function available from both C and C++ code. However,
we observed that we never called this function from C code and that the
'printf' symbol conflicts with the libc. Hence, we turned 'printf'
into a C++ symbol residing in the 'Genode' namespace.
Operating-system services and libraries
#######################################
This section documents the new features and changes affecting
the 'os' repository.
New Features
============
Device-driver framework for C device drivers
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Genode's base API features everything needed to create user-level device
drivers. For example, the 'os' repository's PS/2 input driver and the
PCI bus driver are using Genode's C++ base API directly. However, most of
today's device drivers are written in C. To ease the reuse of existing
drivers on Genode, we have introduced a C API for device drivers into
Genode's 'os' repository. The API is called DDE kit (DDE is an acronym
for device-driver environment) and it is located at 'os/include/dde_kit'.
The DDE kit API is the result of long-year experiences with porting device
drivers from Linux and FreeBSD to custom OS environments. The following
references are the most significant contributions to the development of
the API.
;
Christian Helmuth created the initial version of the Linux device-driver
environment for L4. He describes his effort of reusing unmodified sound
drivers on the L4 platform in his thesis
[https://os.inf.tu-dresden.de/papers_ps/helmuth-diplom.pdf - Generische Portierung von Linux-Gerätetreibern auf die DROPS-Architektur].
;
Gerd Griessbach approached the problem of re-using Linux USB drivers
by following the DDE approach in his diploma thesis
[https://os.inf.tu-dresden.de/papers_ps/griessbach-diplom.pdf - USB for DROPS].
;
Marek Menzer adapted Linux DDE to Linux 2.6 and explored the DDE
approach for block-device drivers in his student research project
[https://os.inf.tu-dresden.de/papers_ps/menzer-beleg.pdf - Portierung des DROPS Device Driver Environment (DDE) für Linux 2.6 am Beispiel des IDE-Treibers ]
and his diploma thesis
[https://os.inf.tu-dresden.de/papers_ps/menzer-diplom.pdf - Entwicklung eines Blockgeräte-Frameworks für DROPS].
;
Thomas Friebel generalized the DDE approach and introduced the DDE kit
API to enable the re-use of device driver from other platforms than
Linux. In particular, he experimented with the block-device drivers of
FreeBSD in his diploma thesis
[https://os.inf.tu-dresden.de/papers_ps/friebel-diplom.pdf - Übertragung des Device-Driver-Environment-Ansatzes auf Subsysteme des BSD-Betriebssystemkerns].
;
Dirk Vogt successfully re-approached the port of USB device drivers
from the Linux kernel to L4 in his student research project
[https://os.inf.tu-dresden.de/papers_ps/beleg-vogt.pdf - USB for the L4 Environment].
The current incarnation of the DDE kit API provides the following
features:
* General infrastructure such as init calls, assertions, debug output
* Interrupt handling (attach, detach, disable, enable)
* Locks, semaphores
* Memory management (slabs, malloc)
* PCI access (find device, access device config space)
* Virtual page tables (translation between physical and virtual
addresses)
* Memory-mapped I/O, port I/O
* Multi-threading (create, exit, thread-local storage, sleep)
* Timers, jiffies
For Genode, we have created a complete reimplementation of the DDE kit
API from scratch by fully utilizing the existing Genode infrastructure
such as the available structured data types, core's I/O services,
the synchronization primitives, and the thread API.
[image dde_kit]
Figure [dde_kit] illustrates the role of DDE kit when re-using an
unmodified device driver taken from the Linux kernel. DDE kit translates
Genode's C++ base API to the DDE kit C API. The DDE kit API, in turn, is
used as back end by the Linux driver environment, which translates Linux
kernel interfaces to calls into DDE kit. With this translation in place,
an unmodified Linux device driver can be embedded into the Linux driver
environment. The device API is specific for a class of devices such as
NICs, block devices, or input devices. It can either be used directly as
a function interface by an application that is using the device driver
as a library, or it can be made accessible to external processes via an
RPC interface.
Limitations
-----------
The PCI sub system is not completely implemented, yet.
Alarm API providing a timed event scheduler
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The scheduling of timed events is a recurring pattern found in device
drivers, application frameworks such as Qt4 ('qeventdispatcher'), and
applications. Therefore, we have added a timed event scheduler to the
'os' repository. The new alarm API ('os/include/os/alarm.h') allows
for the scheduling of both one-shot alarms and periodic alarms.
Changes
=======
PS/2 input driver
~~~~~~~~~~~~~~~~~
The original PS/2 driver tried to switch the PS/2 keyboard to
scan-code set 2 and assumed that all modern keyboards support this
mode of operation. However, this assumption was wrong. We observed
that the legacy PS/2 support of some USB keyboards covers only the
emulated (xlate) scan-code set 1 mode. This is also case for the PS/2
emulation in VirtualBox. Therefore, we changed our PS/2 driver to
never touch the keyboard mode but to only detect the current mode
of operation. The driver has now to support both, scan-code set 1 and
scan-code set 2. This change comes along with a slightly more complex
state machine in the driver. Hence, we moved the state machine from
the IRQ handler to a distinct class and changed the control flow of
the driver to fetch only one value from the i8042 PS/2 controller
per received interrupt.
PCI bus driver
~~~~~~~~~~~~~~
Until now, Genode's PCI bus driver was only used for experimentation
purposes. With the forthcoming driver framework however, the PCI bus
driver will play a central role in the system. Therefore, we adapted
the interface of the PCI driver to these requirements. Specifically,
the scanning of the PCI bus can now be performed without constraining
the results by a specific vendor ID.
Nitpicker GUI server
~~~~~~~~~~~~~~~~~~~~
We improved the _output_latency_ of the Nitpicker GUI server by flushing
pixels eagerly and deferring the next periodically scheduled flush.
This change has a positive effect on the responsiveness of the GUI to
user input.
Misc changes
~~~~~~~~~~~~
Prior versions of the 'os' repository came with a custom 'os/include/base'
directory with interfaces extending the base API. To avoid confusion
between the 'base' repository and the 'os' repository, 'os'-local API
extensions are now located at 'os/include/os'. This way, the folder
prefix of include statements indicates well from which repository the
included header files comes from.
C runtime
#########
Most of existing libraries rely on the presence of a C library. For
making the reuse of this software on Genode possible, we have now
made a complete C library available for Genode. It comes as a separate
source-code repository called 'libc' and is based on the code of FreeBSD.
The original code is available at the official FreeBSD website.
:FreeBSD website:
[https://www.freebsd.org/developers/cvs.html]
Our libc port comprises the libraries 'gdtoa', 'gen', 'locale', 'stdio',
'stdlib', 'stdtime', 'string', and 'msun'. Currently, it supports the
x86 architecture. Support for other architectures is planned as future
addition. At the current stage, our back end is very basic and most of
its functions are dummy stubs. We used Christian Prochaska's forthcoming
Genode port of Qt4 as test case and successfully used the new libc as
foundation for building graphical Qt4 applications. We will further
extend the back end in correspondence to the growing feature set of the
Genode OS framework.
:Usage:
To use the libc in your application, just add 'libc' to the 'LIBS'
declaration in your build-description file. This declaration will make
the libc headers available for the include path of your target and link
the C library. When building, make sure that the 'libc' repository is
included in your build configuration ('etc/build.conf').
:Limitations:
The current version of the C library is not thread-safe. For most
string and math functions, this is not a problem (as these functions
do not modify global state) but be careful with using more complex
functions such as 'malloc' from multiple threads. Also, 'errno' may
become meaningless when calling libc functions from multiple threads.
We have left out the following files from the Genode port of the
FreeBSD libc: gdtoa 'strtodnrp.c' (gdtoa), 'getosreldate.c' (gen),
'strcoll.c', 'strxfrm.c', 'wcscoll.c', 'wcsxfrm.c' (string),
's_exp2l.c' ('msun').
The current back end is quite simplistic and it may help you to revisit
the current state of the implementation in the 'libc/src/lib/libc'
directory. If one of the functions in 'dummies.c' is called, you will
see the debug message:
! "<function-name> called, not yet implemented!"
However, some of the back-end function implemented in the other files
have dummy semantics but have to remain quiet because they are called
from low-level libc code.
Build infrastructure
####################
Build-directory creation tool
=============================
Because we think that each Genode developer benefits from knowing the
basics about the functioning of the build system, the manual creation of
build directories is described in Genode's getting-started document.
However, for regular developers, creating build directories becomes a
repetitive task. Hence, it should be automated. We have now added a
simple build-directory creation tool that creates pre-configured build
directories for some supported platforms. The tool is located at
'tool/builddir/create_builddir'. To print its usage information, just
execute the tool without arguments.
Improved linking of binary files
================================
For linking binary data, binary file have to be converted to object
files. Over the time, we have used different mechanisms for this
purpose. Originally, we used 'ld -r -b binary'. Unfortunately, these
linker options are not portable. Therefore, the mechanism was changed
to a 'hexdump' and 'sed' magic that generated a C array from binary data.
This solution however, is complicated and slow. Now, we have adopted
an idea of Ludwig Hähne to use the 'incbin' directive of the GNU
assembler, which is a very clean, flexible, and fast solution.
Lib-import mechanism
====================
Libraries often require specific include files to be available at the
default include search location. For example, users of a C library
expect 'stdio.h' to be available at the root of the include search
location. Placing the library's include files in the root of the
default search location would pollute the include name space for
all applications, regardless if they use the library or not. To
keep library-include files well separated from each other, we have
enhanced our build system by a new mechanism called lib-import.
For each library specified in the 'LIBS' declaration of a build
description file, the build system incorporates a corresponding
'import-<libname>.mk' file into the build process. Such as file
defines library-specific compiler options, in particular additional
include-search locations. The build system searches for lib-import
files in the 'lib/import/' subdirectories of all used repositories.
Using 'ar' for creating libraries
=================================
The previous versions of Genode relied on incremental linking ('ld -r')
for building libraries. This approach is convenient because the linker
resolves all cross-dependencies between libraries regardless of the
order of how libraries are specified at the linker's command line.
However, incremental linking prevents the linker from effectively
detecting dead code. In contrast, when linking '.a' files, the linker
detects unneeded object files. Traditionally, we have only linked our
own framework containing no dead code. This changed with the new 'libc'
support. When linking the 'libc', the presence of dead code becomes
the normal case rather than the exception. Consequently, our old
incremental-linking approach produced exceedingly large binaries
including all functions that come with the 'libc'. We have now adopted
the classic 'ar' mechanism for assembling libraries and use the linker's
'start-group' 'end-group' feature to resolve inter-library-dependencies.
This way, dead code gets eliminated at the granularity of object files.
In the future, we will possible look into the '-ffunction-sections' and
'-gc-sections' features of the GNU tool chain to further improve the
granularity to function level.
If your build-description files rely on custom rules referring to
'lib.o' files, these rules must be adapted to refer to 'lib.a' files
instead.
Misc changes
============
* Added sanity check for build-description files overriding 'INC_DIR'
instead of extending it.
* Restrict inclusion of dependency files to those that actually matter
when building libraries within 'var/libcache'. This change significantly
speeds up the build process in the presence of large libraries such as
Qt4 and libc.
* Added rule for building 'cpp' files analogously to the 'cc' rule.
Within Genode, we name all C++ implementation files with the 'cc'
suffix. However, Qt4 uses 'cpp' as file extension so we have to
support both.
* Build-description files do no longer need the declaration
'REQUIRES = genode'. Genode's include search locations are now
incorporated into the build process by default.
Applications
############
This section refers to the example applications contained in Genode's
'demo' repository.
We have enhanced the _Scout_widgets_ as used by the launchpad and the
Scout tutorial browser to perform all graphical output double-buffered,
which effectively eliminates drawing artifacts that could occur when
exposing intermediate drawing states via direct (unbuffered) output.
Furthermore, we have added a way to constrain the maximum size of
windows to perform pixel-buffer allocations on realistic window sizes.
Both launchpad and Scout can now start child applications. In Scout
this functionality is realized by special "execute" links. We have
generalized the underlying application logic for creating and
maintaining child processes between both applications and placed
the unification into a separate 'launchpad' library.
We have replaced the default document presented in Scout with an
_interactive_walk-through_guide_ explaining the basic features of Genode.
The document uses the new "execute" link facility to let the user start
a launchpad instance by clicking on a link.
Platform-specific changes
#########################
Genode used to define _fixed-width_integer_types_ in a file 'stdint.h'
placed in a directory corresponding to bit-width of the platform, for
example 'include/32bit/stdint.h'. When building for a 32bit platform,
the build system included the appropriate directory into the
include-search path and thereby made 'stdint.h' available at the root
of the include location. Unfortunately, this clashes with the 'stdint.h'
file that comes with the C library. To avoid conflict with libc header
files, we moved the definition of fixed-width integer types to
'32bit/base/fixed_stdint.h'.
For the L4/Fiasco version of Genode, there existed some x86-specific
header files that did not specifically depend on L4/Fiasco, for example
atomic operations. Because these files are not L4/Fiasco-specific and
may become handy for other platforms as well, we moved them to the
generic 'base' repository.
Linux 32bit
===========
:Dissolving Genode's dependency from the glibc:
The port of the C runtime to Genode posed an interesting challenge to
the Linux version of Genode. This version used to rely on certain
functions provided by the underlying glibc:
* For creating and destroying threads, we used to rely on POSIX threads
as provided by the 'pthread' library
* The lock implementation was based on the POSIX semaphore functions
'sem_init', 'sem_wait', and 'sem_post'
* Shared memory was realized by using files ('open', 'close',
'ftruncate') and the 'mmap' interface
* Starting and killing processes was implemented using 'fork' and 'kill'
* Inter-process communication used the glibc's socket functions
For our custom C runtime, we want to override the glibc functionality
with our own implementation. For example, we want to provide the 'mmap'
interface to a Genode application by implementing 'mmap' with
functions of our base API. On Linux, however, this base API, in turn,
used to rely on 'mmap'. This is just an example. The problem applies
also for the other categories mentioned above. We realized that we cannot
rely on the glibc on one hand but at the same time replace it by a custom
C runtime (in fact, we believe that such a thing is possible by using
awkward linker magic but we desire a clean solution). Consequently, we
have to remove the dependency of Genode from the glibc on Linux. Step
by step, we replaced the used glibc functions by custom Linux system-call
bindings. Each binding function has a prefix 'lx_' such that the symbol
won't collide with 'libc' symbols. The new bindings are located at the file
'base-linux/src/platform/linux_syscalls.h'. It consist of 20 functions,
most of them resembling the original interface ('socket', 'connect',
'bind', 'getsockname', 'recvfrom', 'write', 'close', 'open', 'fork',
'execve', 'mmap', 'ftruncate', 'unlink', 'tkill', 'nanosleep').
For other functions, we simplified the semantics for our use case
('sigaction', 'sigpending', 'sigsetmask', 'create_thread'). The most
noteworthy changes are the creation and destruction of threads by
directly using the 'clone' and 'tkill' system calls, and the lock
implementation. Because we cannot anymore rely on the convenience of
using futexes indirectly through the POSIX semaphore interface, we
have adopted the simple locking approach that we already use for the
L4/Fiasco version. This lock implementation is a simple sleeping
spinlock.
:Compromises:
The introduction of custom Linux system-call bindings for Genode has
several pros and cons. With this change, The Linux version of Genode is
not anymore easy to port to other POSIX platforms such as the Darwin
kernel. For each POSIX kernel used as Genode platform, a custom
implementation of our system-call bindings must be created. The
original POSIX variant could still be reanimated, but this version
would inherently lack support for Genode's C runtime, and thus would
have limited value. A positive side effect of this solution, however,
is that 'linux_syscalls.h' documents well the subset of the Linux'
kernel interface that we are actually using.
The replacement of POSIX semaphores with sleeping spinlocks decreases
locking performance quite significantly. In the contention case, the
wakeup from sleeping introduces a high latency of up to one millisecond.
Furthermore, fairness is not guaranteed and the spinning produces a bit
of system load. If this approach turns out to become a serious performance
bottleneck, we will consider creating custom bindings for Linux' futexes.
L4/Fiasco
=========
The concepts of _RM_faults_ and _managed_dataspaces_ as described in
Section [Base framework], had been implemented into the L4/Fiasco
version of core. Although the introduction of these concepts involved
only minimal changes at the API level, the required core-internal
changes had been quite invasive, affecting major parts of the pager
and RM-session implementations.
Prior versions of the L4/Fiasco version of core did not implement
the _cancel-blocking_mechanism_ as specified by the CPU-session API.
The missing implementation resulted in lock-ups when destructing a
thread that blocks for lock. With the new implementation based on
L4/Fiasco's inter-task ex-regs system call, such threads can now
be gracefully destructed.

460
doc/release_notes/09-02.txt Normal file
View File

@@ -0,0 +1,460 @@
==============================================
Release notes for the Genode OS Framework 9.02
==============================================
Genode Labs
Summary
#######
Whereas the focus of the previous release 8.11 was the refinement of
Genode's base API and the creation of the infrastructure needed to build
real-world applications, the release 9.02 is focused on functional
enhancements in two directions. The first direction is broadening the
number of possible base platforms for the framework. At present, most
microkernels bring along a custom user land, which is closely tied to the
particular kernel. Our vision is to establish Genode as a common ground for
developing applications, protocol stacks, and device drivers in such a way
that the software becomes easily portable among different kernels. This
release makes Genode available on the L4ka::Pistachio kernel. Hence,
software developed with the Genode API can now run unmodified on
Linux/x86, L4/Fiasco, and L4ka::Pistachio. In the second direction, we
are steadily advancing the functionality available on top of Genode. With
this release, we introduce a basic networking facility and support for
native Qt4 applications as major new features. Thanks to Genode's
portability, these features become automatically available on all
supported base platforms.
Our original plan for the release 9.02 also comprised the support of a
Linux-on-Genode (para-)virtualization solution. Initially, we intended to
make [https://os.inf.tu-dresden.de/L4/LinuxOnL4/ - L4Linux] available on
the L4/Fiasco version of Genode. However, we identified several downsides
with this approach. Apparently, the development of the officially available
version of L4/Fiasco has become slow and long-known issues remain unfixed.
L4Linux, however, is closely tied to L4/Fiasco and the L4 environment. For
us at Genode Labs, maintaining both a custom port of L4Linux for Genode
and L4/Fiasco by ourself in addition to developing Genode is unfeasible.
In contrast, the Pistachio kernel features more advanced options for
virtualization ([http://l4ka.org/projects/virtualization/afterburn/ - Afterburner]
and VT support) that we want to explore. Furthermore, there exists another
version of L4Linux called OKLinux for the OKL4 kernel developed at
[http://ok-labs.com - OK-Labs], which is very interesting as well.
Therefore, we decided against an ad-hoc solution and deferred this feature
to the next release. [https://genode.org/about/road-map - See our updated road map...]
Major new Features
##################
Genode on L4ka::Pistachio
=========================
From the very beginning, the base API of the Genode OS Framework was
designed for portability. We put a lot of effort into finding API
abstractions that are both implementable on a wide range of kernels and as
close to the hardware as possible to keep the abstraction overhead low. For
this reason, we developed the framework in parallel for the Linux kernel and
the L4/Fiasco kernel. To validate our claim that Genode is highly portable,
Julian Stecklina ported the framework to another member of the L4 family,
namely the [http://l4ka.org/projects/pistachio/ - L4ka::Pistachio kernel].
This high-performance kernel implements the latest official L4 API called
L4.x2 and has a number of advanced features such as multi-processor support
and virtualization support.
After Julian successfully created the first Pistachio version of Genode,
we successively refined his work and conducted further unifications among
the platform-dependent code for the different kernels. The result of this
effort is included in this release and comes in the form of the
'base-pistachio' source-code repository.
;Interesting technical notes:
* The IRQ handling on Pistachio is slightly different from L4/Fiasco.
On L4/Fiasco, an IRQ becomes unmasked only when the user-level IRQ
handler thread blocks for an IRQ by issuing an IPC call to the
kernel's corresponding IRQ thread. In contrast, Pistachio unmasks an
IRQ as soon as the user-level IRQ handler associates itself with an
IRQ. Thus, an IRQ message may occur not only when the user-level IRQ
handler blocks for any IRQ but anytime. In particular, IRQ messages
may interfere with the IRQ handler's IPC communication with other
threads. To ensure that IRQ messages do only occur when expecting
them, we lazily associate the IRQ handler thread to the IRQ the
first time we wait for an IRQ and issue an unmasking IPC call
subsequent times.
* Genode provides a mechanism for gracefully destructing threads that
are in a blocking state, for example waiting for an IPC message.
Such a thread may hold locks or other resources that would not
get properly freed when just killing the thread by force. Therefore,
Genode provides a way to issue the cancellation of a blocking
operation by another thread (e.g., the thread calling the destructor).
Once, a blocking operation got canceled, a C++ exception
('Blocking_canceled') is raised such the thread can fall back into
a defined state and then be destructed. On L4ka::Pistachio, we use
Pistachio's pager-exchange-registers feature in combination with
the user-defined UTCB handle for cancelling blocking operations and
detecting cancellations. The interesting code bits can be found in
'src/base/ipc/ipc.cc', 'src/base/lock/lock.cc',
'src/core/platform_thread.cc', and in the Pistachio-specific
timer-service implementation.
* During the refinement of the Pistachio version, we were able to further
generalize code that was previously specific for L4/Fiasco and
L4ka::Pistachio respectively. Now, the platform-specific code comprises
less than 3,000 lines of code (LOC) for L4/Pistachio, circa 2,000 LOC
for L4/Fiasco, and circa 1,000 LOC for Linux. Hence, we expect that
porting the framework to further kernels is possible at reasonable
engineering costs.
:Current limitations:
* The current version does not use superpages (4M mappings) because we
experienced problems with mapping 4K pages out of 4M pages. This is an
issue that we like to investigate further because using 4M mappings
would improve the boot time and reduce the kernel-memory usage.
* Currently, we use a simple sleeping spinlock for synchronization, which
is not optimal for several reasons. There are no fairness guarantees,
the spinning consumes CPU time, and threads that got blocked in the
contention case are woken up at the coarse granularity of the kernel's
timer tick, which is typically one millisecond.
* Nested RM sessions as introduced as an experimental feature in the
Genode release 8.11 are not yet supported.
:Further details:
You can find further technical details and usage instructions at this
dedicated [https://genode.org/documentation/platforms/pistachio - page].
Qt4 on Genode
=============
The minimalism of the Genode OS Framework with regard to its code
complexity raised the question of whether this framework is feasible
for hosting real-world applications and widely used runtime environments.
Christian Prochaska took the challenge to port Trolltech's Qt4 application
framework, which serves as the basis for the popular KDE desktop, to Genode.
Because Christian started his work more than a year ago at a time when no
C library was available on Genode, several intermediate steps were needed.
The first step was the integration of the Qt4 tools such as the meta-object
compiler (moc) and resource compiler properly into the our build systion.
With the tools in place, the Linux version of Genode came to an advantage.
In this environment, a Genode application is able to use glibc functionality.
So the problem of a missing C library could be deferred and Christian was
able to focus on interfacing Qt with the existing Genode services such as
the Nitpicker GUI sever. Next, the glibc dependencies were successively
replaced by custom implementations or simple dummy stubs. Thereby, all
needed functionalities such as timed semaphores and thread-local storage
had to be mapped to existing Genode API calls. Once, all glibc dependencies
had been dissolved, Qt could be compiled for the L4/Fiasco version.
Since a C library has become available in Genode 8.11, we were able to
replace Christian's intermediate stub codes with a real C library. We also
utilize recently added features of Genode such as its alarm framework to
simplify the Qt4 port. Furthermore, we were able to remove all
platform-specific bits such that the Qt4 port has now become completely
generic with regard to the underlying kernel. Qt4 can be executed on Linux,
L4/Fiasco, and L4ka::Pistachio without any changes. Figure [qt4_screenshot]
shows a screenshot of Qt's Tetrix example running side-by-side with native
Genode applications.
[image qt4_screenshot]
:Current state:
* The Qt4 port comes in the form of a source-code repository, which contains
all Qt source codes, and some example programs such as Tetrix. You can
download the Qt4 repository as a separate archive at the download page of
the Genode release 9.2. For the next release, we plan to separate the
Genode-specific parts from Qt original code and make the Genode-specific
parts a regular component of the Genode main line.
* The Qt4 port consists of Qt's Core library, GUI library, Script library,
XML library, and the UI tools library. Other libraries such as Webkit
are not ported yet.
* This first version of Qt4 on Genode is not to be considered as stable.
There are several known issues yet to be addressed. In particular,
the 'QEventDispatcher' is still work in progress and not fully stabilized.
* Because, we use to statically link programs, the binaries of Qt
applications are exceedingly large. For example the Tetrix binary is
100MB including debug information and 11MB in the stripped form. For
employing Qt on Genode at a larger scale, Genode should be enhanced with
shared-library support.
Networking
==========
With Genode 8.11, we introduced the Device-Driver-Environment Kit (DDE Kit)
API, which is a C API specifically designed for implementing and porting
device drivers written in plain C. We have now complemented DDE Kit with an
environment for executing Linux device drivers natively on Genode. This
library is called 'dde_linux26' and contained in our new 'linux_drivers'
source-code repository. The environment consists of several parts, which
correspond to the different sub systems of the Linux kernel 2.6, such as
'arch', 'drivers', 'kernel'.
The first class of device-drivers supported by DDE Linux 2.6 is networking.
At the current stage, the DDE Linux network library comprises general
network-device infrastructure as well as an exemplary driver for the PCnet32
network device.
Based on this library, we have created a basic TCP/IP test utilizing the
uIP stack, which uses the DDE Linux network library as back end. The test
program implements a basic web server displaying uIP packet statistics.
When executed on Qemu, you can use your host's web browser to connect to
the web server running on Genode:
For booting Genode on L4/Fiasco with the web-server demo, use a GRUB
entry in your 'menu.lst' file as follows.
! title Genode: DDE Linux 2.6 NET on L4/Fiasco
! kernel /fiasco/bootstrap -maxmem=64 -modaddr=0x02000000
! module /fiasco/fiasco -nokd -serial -serial_esc
! module /fiasco/sigma0
! module /genode/core
! module /genode/init
! module /config
! module /genode/timer
! module /genode/pci_drv
! module /genode/test-dde_linux26_net
The first four lines are L4/Fiasco specific. When using L4ka::Pistachio,
the 'menu.lst' entry looks like this:
! title Genode: DDE Linux 2.6 NET on L4/Pistachio
! kernel /pistachio/kickstart
! module /pistachio/x86-kernel
! module /pistachio/sigma0
! module /genode/core
! module /genode/init
! module /config
! module /genode/timer
! module /genode/pci_drv
! module /genode/test-dde_linux26_net
The web-server test requires the PCI bus driver and the timer service.
Therefore, the 'config' file for Genode's init should have following
content:
! <config>
! <start>
! <filename>timer</filename>
! <ram_quota>512K</ram_quota>
! </start>
! <start>
! <filename>pci_drv</filename>
! <ram_quota>512K</ram_quota>
! </start>
! <start>
! <filename>test-dde_linux26_net</filename>
! <ram_quota>16M</ram_quota>
! </start>
! </config>
Now, its time to create an ISO image from all files specified in
the GRUB configuration. For this, the new utility 'tool/create_iso'
becomes handy. The ISO image can then be booted on Qemu using the
following arguments:
! qemu -m 64 -serial stdio -no-kqemu -cdrom <iso-image> \
! -net nic,model=pcnet -net user -redir tcp:5555::80
The '-redir' argument tells qemu to redirect TCP connections with
localhost:5555 to the guest OS at port 80. After having booted
up Genode on Qemu, you can use your host's web browser to access
the web server:
! firefox http://localhost:5555
:Notes about using the TAP version:
* Preparations
* You must be permitted to sudo and have installed the tunctl
utility. Under Debian/Ubuntu execute
! sudo apt-get install uml-utilities
* Create TAP device
! TAPDEV=$(sudo tunctl -b -u $USER)
! sudo /sbin/ifconfig $TAPDEV 10.0.0.1
* setup DHCP server on $TAPDEV and 10.0.0.0/8
* Run qemu
! qemu -m 64 -serial stdio -no-kqemu -cdrom dde.iso \
! -net nic,model=pcnet \
! -net tap,ifname=$TAPDEV,script=no,downscript=no
* Ping
* Cleanup
* Stop DHCP server
* Remove TAP device
! sudo tunctl -d $TAPDEV
Operating-system services and libraries
#######################################
C Runtime
=========
We have replaced the 'malloc' implementation of the original FreeBSD C
library with a custom implementation, which relies on Genode's 'Heap' as
allocator. The FreeBSD libc reserves a default memory pool of 1MB, which
is no problem on FreeBSD because virtual memory is backed lazily with
physical pages on demand. On Genode however, we immediately account the
allocated memory, which implicates high quota requirements even for
applications that use little memory. In contrast, Genode's heap allocates
and accounts its backing store in relatively small chunks of a few KB.
Therefore, the quota accounting for applications is much more in line with
the actual memory usage. Furthermore, our custom 'malloc' implementation
has the additional benefit of being thread safe.
* Added i386-specific parts of gen lib, in particular longjmp, setjmp.
Device-Driver-Environment Kit
=============================
* The DDE Kit uses our alarm framework (located in the 'os' repository) now
rather than its own event-scheduler implementation formerly called 'Tick'.
* We refined the DDE Kit API and reduced the number of custom types. For
example, we removed the custom 'dde_kit_lock_t' and using
'struct dde_kit_lock' instead, and replaced 'dde_kit_thread_t' with
'struct dde_kit_thread'.
Because of the apparent stabilization of the DDE Kit API, we have now added
this API to Genode's official API reference.
[https://genode.org/documentation/api/dde_kit_index - See the documentation of the DDE Kit API...]
PS/2 input driver
=================
We improved the PS/2 keyboard driver by adding missing scan-code translations
for the scan code set 1, in particular the cursor keys.
Applications
############
Launchpad configuration
=======================
Launchpad is a graphical application for interactively starting and killing
programs. It is used for the default demonstration of Genode. By default,
launchpad displays a preconfigured list of programs and their respective
default memory quotas. The user can tweak the memory quota for each entry
with mouse and then start a program by clicking on its name. As an
alternative to using the default list, you can now define the list manually
by supplying a configuration to Launchpad. The following example tells
launchpad to display a list of two launcher entries:
!<config>
! <launcher>
! <filename>sdl_pathfind</filename>
! <ram_quota>10M</ram_quota>
! </launcher>
! <launcher>
! <filename>liquid_fb</filename>
! <ram_quota>10M</ram_quota>
! </launcher>
!</config>
To use this configuration for a Launchpad started via init, you can simply
insert the launchpad configuration into the '<start>' node of the launchpad
entry in init's 'config' file.
Platform-specific changes
#########################
L4/Fiasco
=========
* Raise 'Blocking_canceled' exceptions on canceled IPC calls
32-bit Linux
============
* We continued dissolving the dependency of Genode from the glibc by using
a custom 'getenv' implementation used during process creation.
* By default, we compile now with '-nostdinc' and explicitly specify
'/usr/include' as include search directory only when needed. Previously,
a Genode application, which included a host include file by mistake, has
not raised any compilation error when compiled for the Linux version of
Genode. Now, all Genode platforms behave equally with regard to include
search directories.
* We enforce using the actual compiler's C++ support libraries rather than
the default libraries installed on the host.
Tools and build infrastructure
##############################
Official tool chain
===================
At the download section of our website, we used to provide a crosstool-based
tool chain as pre-compiled binaries. Since we got several requests about
how to build such a tool chain from scratch, we created custom utility for
downloading, building, and installing the official Genode tool chain. You
can find the utility at 'tool/tool_chain'. For usage instructions, just
start 'tool_chain' without arguments. Because this utility is a plain script,
you can follow and verify each step that we use for creating the Genode tool
chain. Currently, this official tool chain is based on binutils 2.18 and
gcc 4.2.4.
As an alternative to installing the tool chain from source, we also
provide pre-compiled binaries at the download section of our website.
[https://genode.org/download/tool-chain - Visit our tool-chain download website...]
For the Linux version of Genode, we still use the host's default gcc
as tool chain. This way, we spare the hassle of downloading and installing
a custom tool chain for somebody who wants to give Genode a quick try.
With this is mind, we have fixes several small issues with gcc 4.3.2:
* Fixed dependency generation for gcc-4.3.2. Older version of gcc used to
append a '.o' dependency at the target of '.d'-files. However, gcc-4.3.2
seems to handle the option '-MT' differently, resulting in a rule that
contains only the '.d' as target. Now, we explicitly specify both the
'.o' file and the '.d' file as target. Consequently, on older gcc
versions, the '.o' file appears twice but that is no problem.
* Fixed assembler issue with the 'fnstsw' instruction in the C library.
This instruction does not accept eax but only ax as argument.
Build-directory creation tool
=============================
We added a rule for creating a pre-configured build directory for the
Pistachio version to our build-directory creation tool
('tool/builddir/create_builddir'). Furthermore, we changed the default
build configuration such that the official Genode tool chain is used for
L4/Fiasco and L4ka::Pistachio.
Build system
============
* Improved clean rule - visit each target directory only once
* Stop the build process on the first error by default, for continuing
the build process depite of an error, you can use the '-i' argument
of make.
* Compiler flags can now be set specific for compiling C and C++ sources.
This is needed because both variants allow different sets of warning
options. The new variables are called 'CC_CXX_OPT' and 'CC_C_OPT'.
ISO image creation tool
=======================
We have created a convenient front end for 'genisoimage', which we
use for testing Genode on Qemu. You can find this ISO-image-creation
tool at 'tool/create_iso'. For usage instructions, simply start the
tool without arguments.

585
doc/release_notes/09-05.txt Normal file
View File

@@ -0,0 +1,585 @@
==============================================
Release notes for the Genode OS Framework 9.05
==============================================
Genode Labs
Shortly after including support for the L4ka::Pistachio kernel in the
previous release, the Genode version 9.05 gives a further evidence of
Genode's high portability by making the framework available on top of
the OKL4 kernel. This kernel is a commercial-grade microkernel
developed by [http://ok-labs.com - Open Kernel Labs]. In Section
[Supporting the OKL4 kernel as new base platform], we elaborate on the
new base platform and summarize the experiences made during our porting
work.
The previous Genode release was accompanied by a source-code archive containing
the initial version of Qt4 for Genode. Our approach is to make the Qt4
framework available for building Genode applications running natively on the
microkernel rather than within a virtualization environment. As advertised in
our [https://genode.org/about/road-map - road map], we have now seamlessly
integrated the Qt4 framework into our mainline source tree. Furthermore, we
have adapted our port to the Qt4 version 4.5.1. Section [Integration of Qt4
into the mainline repository] gives a rough overview of the changes and an
introduction on how to use the Qt4 framework with Genode.
The third major feature of the release is the addition of preliminary USB
support. We have been able to port major parts of Linux' USB infrastructure
to Genode using the DDE Kit introduced in autumn 2008. Section [USB support]
presents an overview about the pursued design and the current state of
implementation.
Section [OKLinux on Genode] outlines our ongoing efforts of running Linux
as a node in Genode's process tree.
Supporting the OKL4 kernel as new base platform
###############################################
The OKL4 kernel developed by Open Kernel Labs started as a fork of the
L4ka::Pistachio kernel. Whereas L4ka::Pistachio remained true to the L4
x.2 specification, OKL4 was subject of major API changes geared towards high
performance on the ARM architecture. OKL4 earned much fame for executing a
user-level variant of Linux (OKLinux) on top the microkernel, which turned out
to be faster than executing Linux natively on the ARM9 architecture. Even
though OKL4 is primary targeted at the ARM architecture, we wanted to go for
the x86 variant because of two reasons. First, there exists the just mentioned
user-level port of Linux for OKL4, which looks like an attractive way to execute
Linux on Genode once Genode runs on OKL4. Second, we think that distributing
Genode in the form of ISO images bootable on plain PC hardware is the best
way to reach the OS community. Therefore, we decided to use OKL4 version 2.1 as
the base for our work. In contrast to later releases, this version supports
both x86 and ARM. The following section reviews the unique features of the
OKL4 kernel from our perspective.
OKL4 viewed from the angle of a Genode developer
================================================
On the kernel-API level, OKL4 has several interesting properties that had been
both welcome and challenging. We want to highlight the following points:
In contrast to prior L4 kernels, OKL4 has *removed wall-clock timeouts* from
the kernel interface. On L4, timeouts were used as arguments for for blocking
IPC operations serving two purposes. First, specifying IPC timeouts allowed the
IPC caller for regaining control over the blocking thread after the specified
time elapsed. This is considered as important in the case that the called
thread misbehaves and never answers the call. However, the problem of choosing
an appropriate timeout was never properly resolved. When dimensioning the
timeout too small, the called thread may miss the timeout just because it had
no chance to be selected by the scheduler in time. Such timeouts rely on the
presumption that there is low load on the system. On the other hand, when
dimensioning the timeout too high, the system will become sluggish when the
called thread misbehaves. For example, a simple GUI server may want to send
input events to its clients with a timeout to be robust against misbehaving
clients that never wait for events. When choosing a timeout too small, chances
are high that an event will occur at a time when the receiver is handling a
previous event. The timeout would trigger and the event would get lost. When
choosing the timeout too large, say 1 second, any misbehaving client could make
the GUI server freeze for 1 second. Therefore, timeouts for regaining control
over a blocked thread seem to be a bad idea. So we welcome their absence in
OKL4. The second use of timeouts is their use as user-level time source. On L4,
sleep is typically implemented as a blocking IPC with a timeout set to the
sleep value. For this purpose, a system built on top of OKL4 has to employ a
user level device driver accessing a timer device. In Genode, we already have a
timer service for this purpose. So we won't miss timeouts at all.
Classical L4 kernels provide two variants of *synchronous IPC*. So called long
IPC could copy any amount of memory from the sending to the receiving address
space. This is complicated operation because either communication partner may
specify communication buffers that contain unmapped pages. Hence, page faults
may occur during long-IPC operations. On L4, page faults, in turn, are handled
by the user land. Not until a user-level pager thread resolves the page fault
by establishing a mapping at the faulting address, the kernel can proceed the
IPC operation. This sounds pretty complicated, and it is. The second IPC
variant is called short IPC. It constrains the transferable payload to CPU
registers. Hence, these IPC operations should only be used for messages with a
payload of a maximum of 2 machine words. Because short IPCs are never touching
user-level memory pages, no page faults can occur.
On OKL4, there is only one IPC operation, which copies payload from the
sender's user-level thread-control block (UTCB) to the receiver's UTCB. An
UTCB is an always-mapped memory region. Hence no page faults can occur during
IPC operations. On Genode, the UTCB size of 256 bytes may become a limitation
when RPC messages get large. For example, session requests may include large
session-argument strings specifying session-constructor arguments. Current
services rely only on a few arguments so the size limitation is not an
apparent problem. But that may change for future services. Furthermore, in
contrast to L4 x.2, OKL4 does not allow for transferring payload other than
plain data. In particular, OKL4 does not support the transfer of memory
mappings via IPC. Removing memory mappings from the IPC operation is a very
good idea. On Genode, only roottask (core) establishes mappings and shared
memory is implemented as a user-level protocol (data spaces). There is no need
to allow arbitrary processes to establish memory mapping via IPC.
The *boot procedure* of OKL4 largely differs from other L4 kernels. This is
attributed to Open Kernel Labs' focus on embedded systems, which mostly rely on
single-image boot loading. OKL4 employs a tool (elfweaver) for creating a
bootable image from a bunch of files and an XML configuration file. Among the
declarations about which processes to be loaded and which policies to enforce,
the configuration file contains platform parameters such as the amount of
physical memory of the machine. This static approach to configure a system is
certainly useful for embedded systems but PC hardware uses to vary a lot. In
this case, evaluating boot-time memory descriptors would be the preferred
solution.
OKL4 introduces kernel support for *user-level synchronization*. Prior L4
kernels facilitated user-level synchronization through a combination of
synchronous IPC operations with either priorities or delayed preemption.
OKL4's mutexes can make the life in the user land much easier. However, we have
not looked into OKL4 mutexes yet.
There does not exist a recursive *map operation* as the source operand of the
map operation is a physical memory descriptor rather than a virtual address in
the mapper's address space. Consequently, this design eliminates the need for
having a recursive unmap operation and thereby, the need to maintain a mapping
data base in the kernel. This is cool because Genode keeps track of the
mappings at the user level anyway (within core). From our perspective, there is
no need to maintain mapping relationships in the kernel. Removing the mapping
database effectively discards a lot of much-discussed problems about how to
manage the mapping database in a clever way.
There exists *no root memory manager* (sigma0). Because the map operation
takes a physical memory descriptor as argument instead of a virtual address
in the mapper's address space. The mapper does not need to have the mapped
page locally mapped within its own address space. In fact, core (as the only
mapper in a Genode system) does only have very little memory mapped locally.
This design accelerates the boot time because there is no need to map each
physical page in core at startup as performed when running core on the other
L4 kernels.
These differences of OKL4 compared with the microkernels already supported
by Genode posed a number of interesting challenges and opportunities. We have
thoroughly documented the process in
[https://genode.org/documentation/articles/genode-on-okl4 - Bringing Genode to OKL4].
Usage
=====
For using Genode with OKL4, please refer to the following dedicated page:
:[https://genode.org/documentation/platforms/okl4 - Genode on the OKL4 microkernel]:
Site about building and using Genode with the OKL4 kernel.
Limitations of the current implementation
=========================================
The current implementation is able to execute the complete Genode demonstration
scenario on OKL4. This means, we can build and destroy arbitrary trees of
processes, have all the needed infrastructure in place to execute user-level
device drivers such as VESA and PS/2, perform inter-process communication
via RPC and shared memory, and have all basic framework API functions available.
We regard the current state as the first functional version. However, there are
the following points that need improvement and are subject to our future work.
:Incomplete timer driver:
On x86, the timer driver should program the PIT to dispatch sleep requests.
However, the I/O ports of the PIT can only by made available to one party in
the system (which naturally would be the timer driver). Unfortunately, there
are some VESA BIOSes around, which try using the PIT directly. The current
version of our VESA driver does not virtualize these accesses. It rather
tries to gain direct access to the I/O ports from core. This would not work
if the timer already uses this device resource. Our plan is to supplement
our VESA driver with a virtual PIT that uses the timer service as back end.
Then we can safely use the PIT by the timer driver.
:Signalling framework not yet implemented:
We have not yet implemented Genode's API for asynchronous notifications
in the OKL4 version. In fact, the goal of the initial version of the
OKL4 support was running the default demonstration scenario, which does
not rely on signals. The second and more technical reason is that we
consider exploiting OKL4's event mechanism for implementing the signalling
API but have not finalized the design. The generic implementation as used
on the other platforms cannot be used on OKL4 because this implementation
utilizes one helper thread per signal transmitter. Within core, each RM
session is a potential signal transmitter, which means that we need to
create a helper thread per process. Unfortunately, by default, OKL4
limits the number of threads within roottask (core) to only 8 threads,
which would impose a severe limit on the number of processes we could
run on OKL4.
:OKL4's kernel mutexes yet to be used:
We have not yet explored the use of mutexes provided by the OKL4 kernel
for implementing Genode synchronization APIs but we rather rely on a
yielding spin lock for now. This has a number of drawbacks such as high
wake-up latencies in the contention case (depending on the scheduling
time slice), no support for priorities, and no fairness. Although it
is a simple and robust solution to start with, we plan to facilitate
the OKL4 kernel feature with our upcoming work.
:Overly simplistic UTCB allocation:
Right now, we allocate a fixed amount of 32 UTCBs per address space and
thereby limit the maximum number of threads per process. In the future,
this limit should be made configurable.
:Managed dataspaces not yet supported:
The support of managed dataspaces relies on the signal API, which is
not yet available for OKL4.
:Message buffers are limited to 256 bytes:
Because OKL4 performs message-based inter-process communication by
copying data between the UTCBs of the communicating threads, the
UTCB size constaints the maximum message size. Therefore, message
must not exceed 256 bytes. This is not a huge problem for the currently
available Genode programs but we can imagine session argument-lists
to become larger in the future.
:Advanced thread functions are incomplete:
Thread functions such as querying registers of remote threads are not yet
implemented.
Integration of Qt4 into the mainline repository
###############################################
Qt4 is a tool kit for developing platform-independent applications. It
comprises a complete platform-abstraction layer and a rich GUI tool kit
widely used for commercial and open-source applications. It is particularly
known as the technical foundation of the KDE project. The previous Genode
release was accompanied by a snapshot of our initial port of Qt4 to Genode. For
the current release, we have turned this proof-of-concept implementation into a
properly integrated part of the Genode mainline development. This enables Qt4
applications to be executed natively on the full range of kernels supported by
Genode.
Usage
=====
We complemented Genode's source tree with the new 'qt4' source-code repository,
which contains the Genode-specific parts of the Qt4 framework. The most
portions for the Qt4 framework are used unmodified and thereby have not been
made part of the Genode source tree. Instead, we fetch the original Qt4 source
code from Trolltech's FTP server. This way, our source tree remains tidy and
neat.
For using Qt4 for your Genode applications, you first need to download and
prepare the original Qt4 source codes and build a few Qt4 tools such as the
meta-object compiler and the resource compiler. The makefile found in the
top-level directory of the 'qt4' repository automates this task:
! make prepare
To include the 'qt4' repository into the Genode build process, just add the
'qt4' directory to the 'REPOSITORIES' declaration of the 'etc/build.conf' file
within your build directory. Make sure that the repositories 'demo' and 'libc'
are included as well. The 'qt4' repository comes with a couple of demo applications.
The 'qt_launchpad' is especially interesting because it makes use of both the
Qt4 framework and the Genode framework in one application.
Features and limitations
========================
The Qt4 port comprises Qt's Core library, GUI library, Script library, XML
library, and the UI tools library.
For using Qt4 on the Linux version of Genode, we recommend using the Genode
tool chain rather than your host's tool chain. Qt4 makes use of a lot of libc
functionality, supplied by Genode's 'libc' repository. However, on Linux we
still link against your host's libc. This becomes a problem if your host
compiler's C++ support code references libc functionality that is not part of
Genode's libc. Thereby the linker will silently create references to glibc
symbols, making both libraries collide. So if using Qt4, we recommend using the
Genode tool chain:
:[https://genode.org/download/tool-chain]:
Information about downloading and using the Genode tool chain
USB support
###########
This release introduces the first fragments of USB support to Genode, taking
the USB human-interface device (HID) class as starting point. With this work,
we follow our approach of reusing unmodified Linux device drivers executed
within a device-driver environment called DDE Linux. In the previous release,
we already utilized this approach for realizing basic networking on Genode.
With this release, we complement DDE Linux with support required by USB
drivers. We are grateful for being able to base our implementation on the
excellent foundation laid by Dirk Vogt. He described his work in
[https://os.inf.tu-dresden.de/papers_ps/beleg-vogt.pdf - USB for the L4 environment].
For USB HID support, we added the Linux USB and input subsystems to the DDE
Linux 2.6 framework. Besides the 'dde_linux26/net.h' API for network drivers
added in Genode 9.02, the current version also includes APIs for input
('dde_linux26/input.h') and USB ('dde_linux26/usb.h'). We intend these
interfaces to mature towards generic driver-library APIs in the future. For
example, BSD-based drivers shall transparently provide the same functionality
as the current Linux drivers, which permits the simple reuse of driver server
implementations.
[image usb_current]
Image [usb_current] illustrates the current implementation of the USB-based
human-interface device (HID) driver. In this monolithic setup, all parts of the
USB stack and the device API are executed within one address space. These parts
are
* Input server glue code
* HID driver and input subsystem
* Core functions for management of USB request buffers (URBs),
attached devices, and registered drivers
* Host controller drivers for UHCI, OHCI, and EHCI
[image usb_aspired]
We regard this as an intermediate step towards our goal to decompose the USB
stack. Image [usb_aspired] shows our aspired design. In this design, the
USB server and one or more USB gadget drivers run in dedicated address spaces.
The USB server provides two interfaces called USB session interface and USB
device interface. A USB session interface corresponds to a virtual root hub,
from which USB devices can be queried. The client of the USB session interface
is usually an USB gadget driver that uses the USB device interface. Because
this interface is used for transferring the actual payload at a potentially
high bandwidth, it is based on shared memory and signals. The USB server
consists of the following components:
* USB server glue code
* Virtual USB device driver managing all attached devices
* Core functions including hardware hub management
* Host controller drivers
The USB server presents a virtual USB hub to each USB gadget driver. Such
a driver consists of:
* Device interface, e.g., input server glue code
* Gadget driver, e.g., HID driver and input subsystem
* Core functions
* Virtual host controller
* USB client glue code
The HID driver uses the USB session API to monitor ports of its virtual root
hub and submit URBs to attached devices. The session interface facilitates the
signalling framework for event notification and a shared-memory dataspace for
URB transmission.
The 'os' repository already contains the USB session and USB device interfaces.
However, the decomposition is not yet in a functional state.
:Current limitations:
The current monolithic implementation of the USB HID service can already be
used as a replacement of the PS/2 driver. However, both drivers cannot be used
at the same time, yet. To enable the use of both USB HID and PS/2, we plan to
create a further component that merges multiple streams of input events and
passes the result to the GUI server.
OKLinux on Genode
#################
According to our road map, we pursued the goal to run Linux as a node in
Genode's process tree. We explored two approaches:
:Reanimating the Afterburner project conducted by the [http://l4ka.org - L4Ka group]:
This approach is the result of the L4Ka groups's long-year experience with
manually supporting L4Linux on top of the L4ka::Pistachio kernel. Because of
the high costs of maintaining the paravirtualized Linux kernel, a
semiautomatic paravirtualization technique was created. According to the
impressive results presented in
[http://www.l4ka.org/l4ka/publ_2006_levasseur-ua_soft-layering.pdf - Pre-Virtualization: Soft Layering for Virtual Machines],
this approach is able to drastically reduce maintenance costs while retaining
good performance. Furthermore, the approach was applied not only to Linux
running on the L4 kernel but also for using Xen or Linux as underlying
host operating systems.
:Porting the OKL4-specific version of L4Linux to Genode:
Open Kernel Labs maintain a custom version of L4Linux that runs on OKL4. This
version is mostly referred to as OKLinux aka Wombat. Since Genode can now use OKL4
as base platform, the reuse of OKLinux in combination with Genode has become
a feasible option.
Both approaches have pros and cons. Whereas Afterburner is a intriguing
approach, this project seems to be stalled. It relies on a rather old tool
chain, and recent Linux features such as thread-local storage support are not
considered, yet. To pick up this solution for Genode will require us to fully
understand the mechanisms and the code. So we consider this as a mid-term
solution. In short term, running OKLinux on Genode is more feasible. We were
already able to create a prototype version of OKLinux running on Genode. This
version starts up the kernel including all Linux kernel threads, mounts the
boot partition supplied as memory image, and starts the init process. The
engineering costs had been rather low. We replaced the Iguana user land
libraries as originally used by Wombat by a Genode-specific reimplementation to
keep our manual adaptions of the Linux kernel code as small as possible.
Our custom reimplementation of the needed Iguana APIs consists of less than
1,000 lines of code (SLOC). The diff for our changes to the OKLinux kernel code
comprises less than 1,000 lines. We plan to make a snapshot of this prototype
publicly available soon.
Operating-system services and libraries
#######################################
Nitpicker GUI server
====================
We optimized the performance of the 'refresh' call, which updates all views of
a session, which display a given buffer portion. The new implementation restricts
the redraw operations to the fragment of each view that displays the specified
buffer portion. The performance improvement becomes most visible when updating
only small parts of a buffer.
USB session interface
=====================
Genode's emerging USB support introduces two new interfaces to the 'os' repository,
which are called USB session and USB device.
An _USB_session_ is a virtual USB bus with just one root hub with 'MAX_PORTS'
downstream ports. The client of such as session submits USB request blocks
(URBs) and is, in turn, informed about port changes on the root hub as well as
request completion. Connected USB devices can be referenced by USB device
capabilities and are associated with one port at the virtual root hub on
server side.
An _USB_device_ references a hardware device connected to a virtual USB bus's
root hub. The device capability enables the client to send USB request
blocks to the hardware device.
Input interface
===============
We updated the key codes of the input interface in response to recent changes
of Linux' 'dev/event' definitions.
VESA driver
===========
Until now, there existed different processes that tried to access the PCI bus
via I/O ports, in particular the VESA framebuffer driver and the PCI bus
driver.
Since core enforces that each I/O port can only be assigned exclusively to one
component in the system, multiple processes that need access to the same I/O
ports cannot run at the same time. For our default demonstration scenario, we
had been able to allow the VESA driver to use the PCI I/O ports because nobody
else needed them. However, our growing base of device drivers relies on the
PCI bus driver. To be able to use the VESA driver together with other drivers,
we virtualized the access to the PCI bus from within the VESA driver.
Our current PCI virtualization code is pretty limited. The VESA driver sees a
virtual PCI bus with only the VGA card attached. For now, we only allow reading
the PCI configuration space of this device, but not writing to it. Apparently,
this simple approach is sufficient to run the VESA BIOS of Qemu. However, other
VESA BIOS implementations may need further access to the PCI device's
configuration space. For example, for querying the size of a PCI resource,
write access to base address registers is required. In such an event, the VESA
driver will print a message about the missing virtualization feature:
! writing data register not supported
If you see such a message, we are very interested to see your log output such
that we can enhance our PCI virtualization code as needed. Please contact us!
Base framework
##############
In the process of bringing Genode to the OKL4 kernel, we have generalized much
of former platform-specific code:
* The initialization of C++ exception handling has now become part of the
generic 'cxx' library in the 'base' repository. All platforms except
Linux are using this generic library now.
* The 'server' library used to contain a platform-specific part that
implemented the 'manage' function of a 'Server_entrypoint'. The
generalized version of this library is now being used on all platforms
other than Linux.
* We unified core-internal interfaces and their implementations such as
'Dataspace_component', 'Cap_session_component', 'Rm_session_component',
and 'Irq_session_component'. The result has become part of the 'base'
repository.
* On OKL4, threads need to execute small startup code for querying their
own thread IDs. Therefore, we have extended the 'Thread_base' interface
with a platform-specific hook function called '_thread_bootstrap'.
* The types defined in 'base/native_types.h' had been complemented by a
new 'Native_thread_id' type. This type is exclusively used by core and the
framework libraries. For using the Genode API, this type is meaningless.
* For the 64bit support, we slightly refined the interfaces of some utility
template functions in 'util/misc_math.h'. Furthermore, parts of the generic
marshalling code of the IPC framework needed refinement, but no API changes
were needed.
Linux-specific changes
######################
Adaptation to 64 bit
====================
Because most Genode developers tend to work with the Linux version of Genode,
supporting 64-bit Linux becomes increasingly important. With the current release,
we start to officially support 64-bit Linux as base platform. This comes
along with the following changes:
* We replaced the 'spec-x86.mk' file with new 'spec-x86_32.mk' and 'spec-x86_64.mk'
files. The default version of 'base-linux/etc/specs.conf' automatically
chooses the right spec file according to the output of 'uname -m'. Therefore,
output of the build processes matches your host architecture. This behaviour
can be changed by placing a customized 'spec.conf' file in your build directory's
'etc/' subdirectory.
* We added type definitions for 64-bit-specific fixed-size integers in the form
of a 64-bit-specific 'fixed_stdint.h' file.
* Because using 64 bit instead of 32 bit changes the payload size of RPC
messages, we had to adjust several message buffers such as 'Ram_session_client'
and 'Input::Session_client', and adapted the used stack sizes.
* Towards the goal of completely dissolving Genode's dependency on the Linux' glibc,
we implemented custom system-call bindings. Apparently, Linux' syscall interface
differs between 32 bit and 64 bit. For example, the 32-bit version handles
all socket-related calls via a compound 'socketcall' whereas the 64-bit
version uses distinct syscalls. Another difference is the handling of the
'mmap' syscall and different behaviour of 'tkill'. The latter problem was
resolved by replacing 'tkill' with 'tgkill' and setting the thread-group
argument of the corresponding PID. Therefore, a 'Native_thread_id' on Linux
now comprises both the TID and the PID.
* The 'Platform_env' on Linux contains a local implementation of the 'Rm_session'
interface, which uses 'mmap' to attach dataspaces to the process' address
space and maintains the region list in the form of a static array. This array
was dimensioned to 256 entries, which constrained the maximum amount of
usable memory when allocating a lot of small blocks via Genode's heap. Since
the heap allocates backing store at the granularity of only 16KB, the worst
case for reaching this limit was about 4MB. This was OK for our simple test
applications. But for using Qt4, in particular on 64 bit, this has become a
serious limitation. For now, we increased the region limit to 4096 and plan
to replace the static array with a dynamically growing data structure.
Furthermore, we made the heap granularity depend on the actual machine-word
size. Therefore, the heap allocates its backing store in 32KB blocks when
running on 64 bit.
Debugging hooks
===============
On Linux, we use gdb for debugging Genode. This is feasible as long as the
targeted process is running. However, during low-level debugging, we had the
recurring problem of a thread dying shortly after starting up. So we added a hook
for halting a thread at the startup in order to be able to attach gdb to the
thread before it dies. This simple hook lets the thread wait for a key press by
directly calling the 'read' syscall. We also added a simple debug facility for
printing debug messages bypassing Genode's LOG service by calling the 'write'
syscall directly. Both hooks are now part of the Linux version of the 'env'
library (see 'base-linux/src/base/env/debug.cc'). Note that these hooks are not
part of the Genode API. There exists no header file.

0
doc/release_notes-09-08.txt → doc/release_notes/09-08.txt
View File

1016
doc/release_notes/09-11.txt Normal file
View File

File diff suppressed because it is too large Load Diff

1224
doc/release_notes/10-02.txt Normal file
View File

File diff suppressed because it is too large Load Diff

1211
doc/release_notes/10-05.txt Normal file
View File

File diff suppressed because it is too large Load Diff

0
doc/release_notes-10-08.txt → doc/release_notes/10-08.txt
View File

871
doc/release_notes/10-11.txt Normal file
View File

@@ -0,0 +1,871 @@
===============================================
Release notes for the Genode OS Framework 10.11
===============================================
Genode Labs
During the past three months, the Genode project was primarily driven by our
desire to create a bigger picture out of the rich set of components that we
introduced over time, in particular over the last year. Looking back at the
progress made since mid 2009, there were many functional additions to the
framework, waiting to get combined. To name a few, we added support for
networking, audio output, real-time priorities, mandatory access control,
USB, ATAPI block devices, Python, hardware-accelerated 3D graphics, Qt4,
the WebKit-based Arora browser, and the paravirtualized OKLinux kernel.
So many wonderful toys waiting to get played with. This is how the idea of
creating [https://genode.org/download/live-cds - the new Genode Live CD] was
born. In the past, Genode was mostly used in settings with a relatively static
configuration consisting of several components orchestrated to fulfill
a few special-purpose functions. Now, the time has come for the next step,
creating one dynamic setup that allows for the selection of different subsystems
at runtime rather than at boot time.
This step is challenging in several ways. First, the processes that form
the base system have to run during the entire time of all demo setups. If
any of those processes contained stability problems or leaked memory, it would
subvert the complete system. Second, the components of all subsystems combined
are far too complex to be loaded into memory at boot time. This would not
only take too long but would consume a lot of RAM. Instead, those components
and their data had to be fetched from disk (CDROM) on demand. Third, because
multiple demo subsystems can be active at a time, low-level resources such as
networking and audio output must be multiplexed to prevent different
subsystems from interfering with each other. Finally, we had to create a
single boot and configuration concept that is able to align the needs of all
demos, yet staying manageable.
Alongside these challenges, we came up with a lot of ideas about how Genode's
components could be composed in new creative ways. Some of these ideas such
as the browser-plugin concept and the http-based block server made it onto
the Live CD. So for producing the Live CD, we not only faced the said
technical challenges but also invested substantial development effort in new
components, which contributed to our overall goal. Two weeks ago, we released
the Live CD. This release-notes document is the story about how we got there.
To keep ourself focused on the mission described above, we deferred the
original roadmap goal for this release, which was the creation of a Unix-like
runtime environment to enable compiling Genode on Genode. This will be the
primary goal for the next release.
Execution environment for gPXE drivers
######################################
Up to now, DDE Linux provided Genode with drivers for hardware devices
ranging from USB HID to WLAN. In preparation of the live CD, we
noticed the demand for support of a broader selection of ethernet
devices. Intel's e1000 PCI and PCIe cards seemed to mark the bottom
line of what we had to support. The major advantage of NIC drivers
from Linux is their optimization for maximum performance. This emerges
a major downside if DDE Linux comes into play: We have to provide all
the nifty interfaces used by the driver in our emulation framework. To
achieve our short-term goal of a great live CD experience, we had to
walk a different path.
[https://gpxe.org/ - gPXE] is a lovely network boot loader / open-source
PXE ROM project and the successor of the famous Etherboot
implementation. Besides support for DNS, HTTP, iSCSI and AoE, gPXE
includes dozens of NIC drivers and applies a plain driver framework.
As we were also itching to evaluate DDE kit and the DDE approach at
large with this special _donator OS_, we went for implementing the
device-driver environment for gPXE (DDE gPXE).
The current version provides drivers for e1000, e1000e, and pcnet
devices. The emulation framework comprises just about 600 lines of
code compared to more than 22,000 LOC reused unmodified from gPXE.
Benchmarks with the PCNet32 driver showed that DDE gPXE's performance
is comparable to DDE Linux.
The gPXE driver environment comes in the form of the new 'dde_gpxe'
repository. For building DDE gPXE, you first need to download and patch
the original sources. The top-level makefile of this repository automates
this task. Just issue:
! make prepare
Now, you need to include the DDE gPXE repository into your Genode
build process. Just add the path to this directory to the
'REPOSITORIES' declaration of the 'etc/build.conf' file within your
build directory, for example
! REPOSITORIES += $(GENODE_DIR)/dde_gpxe
After successful build the DDE gPXE based ethernet driver is located
at 'bin/gpxe_nic_drv'.
On-demand paging
################
In the [https://genode.org/documentation/release-notes/8.11#section-8 - release 8.11],
we laid the foundation for implementing user-level dataspace managers.
But so far, the facility remained largely unused except for managing thread
contexts. This changed with this release.
So what is a user-level dataspace manager and who needs it? In short,
Genode's memory management is based on dataspaces. A dataspace is a
container for memory. Normally, it is created via core's RAM or ROM
services. The RAM service hands out dataspaces containing contiguous
physical memory. After allocating such a RAM dataspace, the creator can
attach the dataspace to its own address space to access the dataspace
content. In addition, it can pass a dataspace reference (called dataspace
capability) to other processes, which, in turn, can than attach the same
dataspace to their local address space, thereby establishing shared memory.
Similarly, core's ROM service hands out boot-time binary data as dataspaces.
For the most use cases of Genode so far, these two core services were the
only dataspace providers needed. However, there are use cases that require
more sophisticated memory management. For example, to implement swapping,
the content of a dataspace must be transferred to disk in a way that
is transparent to the users of the dataspace. In monolithic kernels, such
functionality is implemented in the kernel. But on a multi-server OS
such as Genode, this is no option. Implementing such a feature into
core would increase the trusted computing base of all applications
including those who do not need swapping. Core would need a hard-disk
driver, effectively subverting the Genode concept. Other examples for
advanced memory-management facilities are copy-on-write memory and
non-contiguous memory - complexity we wish to avoid at the root of the
process tree. Instead of implementing such memory management facilities
by itself, core provides a mechanism to let any process manage dataspaces.
This technique is also called user-level page-fault handling.
For the Live CD, we decided to give Genode's user-level page-fault handling
facility a go. The incentive was accessing files stored on CDROM in an
elegant way. We wanted to make the CDROM access completely transparent to
the applications. An application should be able to use a ROM session as
if the file was stored at core's ROM service. But instead of being
provided by core, the session request would be delegated to an
alternative ROM service implementation that reads the data from disk
as needed. Some of the files stored in the CDROM are large. For example,
the disk image that we use for the Linux demo is 160MB. So reading
this file at once and keeping it in memory is not an option. Instead, only
those parts of the file should be read from disk, which are actually
needed. To uphold the illusion of dealing with plain ROM files for
the client, we need to employ on-demand-paging in the CDROM server.
Here is how it works.
# The dataspace manager creates an empty managed dataspace. Core
already provides a tool for managing address spaces called
region manager (RM service). A RM session is an address space,
to which dataspaces can be attached. This is exactly what is
needed for a managed dataspace. So a dataspace manager uses the
same core service to define the layout of a managed dataspace
as is used to manage the address space of a process. In fact,
any RM session can be converted into a managed dataspace.
! enum { MANAGED_DS_SIZE = 64*1024*1024 };
! Rm_connection rm(0, MANAGED_DS_SIZE);
This code creates a RM session with the size of 64MB. This is an empty
address space. A dataspace capability that corresponds to this address
space can then be requested via
! Dataspace_capability ds = rm.dataspace();
# The dataspace capability can be passed to a client, which may
attach the dataspace to its local address space. Because the
managed dataspace is not populated by any backing store, however,
an access would trigger a page fault, halting the execution of
the client. Here, the page-fault protocol comes into play.
# The dataspace manager registers itself for receiving a signal each time
a fault occurs:
! Signal_receiver rec;
! Signal_context client;
! Signal_context_capability sig_cap = rec.manage(client);
! rm.fault_handler(sig_cap);
When an empty part of the managed dataspace is accessed by any
process, a signal is delivered. The dataspace manager can then
retrieve the fault information (access type, fault address) and
dispatch the page fault by attaching a real dataspace at the
fault address of the managed dataspace. In a simple case, the code
looks as follows:
! while (true) {
! Signal signal = rec.wait_for_signal();
! for (int i = 0; i < signal.num(); i++) {
! Rm_session::State state = rm.state();
! ds = alloc_backing_store_dataspace(PAGE_SIZE);
! rm.attach_at(ds, state.addr & PAGE_MASK);
! }
! }
This simple page-fault handler would lazily allocate a page of
backing store memory each time a fault occurs. When the backing
store is attached to the managed dataspace, core will automatically
wake up the faulted client.
# The example above has the problem that the dataspace manager has
to pay for the backing store that is indirectly used by the client.
To prevent the client from exhausting the dataspace manager's memory,
the dataspace manager may choose to use a limited pool of backing
store only. If this pool is exceeded, the dataspace manager can reuse
an already used backing-store block by first revoking it from its
current managed dataspace:
! rm.detach(addr);
This will flush all mappings referring to the specified address
from all users of the managed dataspace. The next time, this
address region is accessed, a new signal will be delivered.
This page-fault protocol has the following unique properties. First,
because core is used as a broker between client and dataspace manager, the
dataspace manager remains completely unaware of the identity of its client.
It does not even need to possess the communication right to the client. In
contrast, all other user-level page-fault protocols that we are aware of
require direct communication between client and dataspace manager. Second,
because dataspaces are used as first-level objects to resolve page faults,
page faults can be handed at an arbitrary granularity (of course, a multiple
of the physical page size). For example, a dataspace manager may decide to
attach backing-store dataspaces of 64K to the managed dataspace. So the
overhead produced by user-level page-fault handler can be traded for the
page-fault granularity. But most importantly, the API is the same across
all kernels that support user-level page fault handling. Thus the low-level
page-fault handling code becomes inherently portable.
Having said that, we have completed the implementation of the described
core mechanisms, in particular the 'detach' facility, for OKL4. The ISO9660
driver as featured on the Live CD implements the 'ROM' interface and
reads the contents of those files from CDROM on demand. It uses a
fixed pool of backing store, operates at a page-fault granularity of
64KB, and implements a simple fifo replacement strategy.
Base framework
##############
There had been only a few changes to the base framework described as
follows.
We unified the core-specific console implementation among all
base platforms and added synchronization of 'vprintf' calls.
The kernel-specific code resides now in the respective
'base-<platform>/src/base/console/core_console.h' files.
We removed the argument-less constructor from 'Allocator_avl_tpl'.
This constructor created an allocator that uses itself for
meta-data allocation, which is the usual case when creating
local memory allocators. However, on Genode, this code is typically
used to build non-memory allocators such as address-space regions.
For these use cases, the default policy is dangerous. Hence, we
decided to remove the default policy.
The 'printf' helper macros have been unified and simplified. The
available macros are 'PINF' for status information, 'PWRN' for warnings,
'PLOG' for log messages, and 'PERR' for errors. By default, the message
types are colored differently to make them easily distinguishable.
In addition to normal messages, there is the 'PDBG' for debugging
purposes. It remains to be the only macro that prints the function name
as message prefix and is meant for temporary messages, to be removed
before finalizing the code.
Genode's on-demand-paging mechanism relies on the signalling framework.
Each managed dataspace is assigned to a distinct signal context.
Hence, signal contexts need to be created and disposed alongside
with managed dataspaces. We complemented the signalling framework
with a 'dissolve' function to enable the destruction of signal
contexts.
Operating-system services and libraries
#######################################
Finished transition to new init concept
=======================================
With the release 10.05, we introduced the
[https://genode.org/documentation/release-notes/10.05#section-0 - current configuration concept of init].
This concept supports mandatory access control and provides flexible
ways for defining client-server relationships. Until now, we maintained
the old init concept. With the current release, the transition to the
new concept is finished and we removed the traditional init.
We retained the support for loading configurations for individual subsystems
from different files but adopted the syntax to the use of attributes.
Instead of
! <configfile>subsystem.config</configfile>
the new syntax is
! <configfile name="subsystem.config"/>
Virtual network bridge (Proxy ARP)
==================================
Since we originally added networking support to Genode, only one program could
use the networking facilities at a time. In the simplest form, such a program
included the network driver, protocol stack, and the actual application. For
example, the uIP stack featured with release 9.02 followed this approach.
In release 9.11 we added the 'Nic_session' interface to decouple the network
driver from the TCP/IP protocol stack. But the 1-to-1 relation between
application and network interface remained. With the current release, we
introduce the 'nic_bridge' server, which is able to multiplex the 'Nic_session'
interface.
The implementation is roughly based on the proxy ARP RFC 1027. At startup, the
'nic_bridge' creates a 'Nic_session' to the real network driver and, in turn,
announces a 'Nic' service at its parent. But in contrast to a network driver
implementing this interface, 'nic_bridge' supports an arbitrary number of
'Nic_sessions' to be opened. From the client's perspective, such a session
looks like a real network adaptor.
This way, it has become possible to run multiple TCP/IP stacks in
parallel, each obtaining a distinct IP address via DHCP. For example,
is has become possible to run multiple paravirtualized Linux kernels
alongside an lwIP-based web browser, each accessing the network via a
distinct IP address.
As a side effect for developing the 'nic_bridge', we created a set
of utilities for implementing network protocols. The utilities are
located at 'os/include/net' and comprise protocol definitions for
ethernet, IPv4, UDP, ARP, and DHCP.
Nitpicker GUI server
====================
Our work on the Live CD motivated several improvements of the Nitpicker
GUI server.
Alpha blending
~~~~~~~~~~~~~~
In addition to nitpicker's plain pixel buffer interface that is compatible
with a normal framebuffer session, each nitpicker session can now have
an optional alpha channel as well as an corresponding input-mask channel
associated. Both the alpha channel and the input mask are contained in the
same dataspace as the pixel buffer. The pixel buffer is followed by
the 8-bit alpha values, which are then followed by the input-mask values.
This way, the presence of an alpha channel does not interfere with the
actual pixel format. Each 8-bit input mask value specifies the user-input
policy for the respective pixel. If the value is zero, user input
referring to the pixel is not handled by the client but "falls through"
the view that is visible in the background of the pixel. This is typically
the case for drop shadows. If the input-mask value is '1', the input
is handled by the client.
With the input-mask mechanism in place, we no longer have a definitive
assignment of each pixel to a single client anymore. In principle, an
invisible client is able to track mouse movements by creating a full-screen
view with all alpha values set to '0' and all input-mask values set to '1'.
Once, the user clicks on this invisible view, the user input gets routed
to the invisible client instead of the actually visible view. This
security risk can be addressed at two levels:
* In X-Ray mode, nitpicker completely disables alpha blending
and the input-mask mechanism such that the user can identify the
client that is responsible for each pixel on screen.
* The use of the alpha channel is a session argument, which is specified
by nitpicker clients at session-creation time. Consequently, this
session argument is subjected to the policy of all processes involved
with routing the session request to nitpicker. Such a policy may permit
the use of an alpha channel only for trusted applications.
_Caution:_ The use of alpha channels implies read operations from
the frame buffer. On typical PC graphics hardware, such operations are
extremely slow. For this reason, the VESA driver should operate in
buffered mode when using alpha blending in Nitpicker.
Tinted views in X-Ray mode
~~~~~~~~~~~~~~~~~~~~~~~~~~
We added support for tinting individual clients or groups of clients
with different colors based on their label as reported at session-creation
time. By using session colors, nitpicker assists the user to tell apart
different security domains without reading textual information. In
addition to the tinting effect, the title bar presents the session
color of the currently focused session.
The following nitpicker configuration tints all views of the launchpad
subsystem in blue except for those views that belong to the testnit
child of launchpad. Those are tinted red.
! <config>
! <policy label="launchpad" color="#0000ff"/>
! <policy label="launchpad -> testnit" color="#ff0000"/>
! </config>
Misc Nitpicker changes
~~~~~~~~~~~~~~~~~~~~~~
We introduced a so-called 'stay-top' session argument, which declares
that views created via this session should stay on top of other views.
This function is useful for menus that should always remain accessible
or banner images as used for Live CD.
Nitpicker's reserved region at the top of the screen used to cover up
the screen area as seen by the clients. We have now excluded this area
from the coordinate system of the clients.
We implemented the 'kill' mode that can be activated by the 'kill' key.
(typically this is the 'Print Screen' key) This feature allows the user
to select a client to be removed from the GUI. The client is not
actually killed but only locked out. The 'kill' mode is meant as an
emergency brake if an application behaves in ways not wanted by the
user.
ISO9660 server
==============
As outlined in Section [On-demand paging], we revisited the ISO9660 server
to implement on-demand-paged dataspaces. It is the first real-world
use case for Genode's user-level page-fault protocol. The memory pool
to be used as backing store for managed dataspaces is dimensioned according
to the RAM assigned to the iso9660 server. The server divides this backing
store into blocks of 64KB and assigns those blocks to the managed dataspaces
in a fifo fashion. We found that using a granularity of 64KB improved the
performance over smaller block sizes because this way, we profit from reading
data ahead for each block request. This is particularly beneficial for
CDROM drives because of their extremely long seek times.
Audio mixer
===========
We added a new *channel synchronization* facility to the 'Audio_out_session'
interface. An 'Audio_out_session' refers to a single channel. For stereo
playback, two sessions must be created. At session-creation time, the
client can provide a hint about the channel type such as "front-left" as
session-construction argument. This design principally allows for supporting
setups with an arbitrary amount of channels. However, those channels must
be synchronized. For this reason, we introduced the 'sync_session' function
to the 'Audio_out_session' interface. It takes the session capability of
another 'Audio_out_session' as argument. The specified session is then
used as synchronization reference.
To reduce the latency when stopping audio replay, we introduced a new *flush*
function to the 'Audio_out_session' interface. By calling this function,
a client can express that it is willing to discard all audio data already
submitted to the mixer.
Furthermore, we improved the audio mixer to support both long-running
streams of audio and sporadic sounds. For the latter use case, low latency
is particularly critical. In this regard, the current implementation is a
vast improvement over the initial version. However, orchestrating the
mixer with audio drivers as well as with different clients (in particular
ALSA programs running on a paravirtualized Linux) is not trivial. In the
process, we learned a lot, which will eventually prompt us to further
optimize the current solution.
Nitpicker-based virtual Framebuffer
===================================
To support the browser-plugin demo, we introduced 'nit_fb', which is a
framebuffer service that uses the nitpicker GUI server as back end. It
is similar to the liquid framebuffer as featured in the 'demo' repository
but in contrast to liquid framebuffer, 'nit_fb' is non-interactive.
It has a fixed screen position and size. Furthermore, it does not
virtualize the framebuffer but passes through the framebuffer portion of
the nitpicker session, yielding better performance and lower latency.
If instantiated multiple times, 'nit_fb' can be used to statically arrange
multiple virtual frame buffers on one physical screen. The size and screen
position of each 'nit_fb' instance can be defined via Genode's configuration
mechanism using the following attributes of the 'nit_fb' config node:
! <config xpos="100" ypos="150"
! width="300" height="200"
! refresh_rate="25"/>
If 'refresh_rate' isn't set, the server will not trigger any refresh
operations by itself.
On the Live CD, each browser plugin instantiates a separate instance of
'nit_fb' to present the plugin's content on screen. In this case, the
view position is not fixed because the view is further virtualized by the
loader, which imposes its policy onto 'nit_fb' - Genode's nested
policies at work!
TAR ROM service
===============
For large setups, listing individual files as boot modules in single-image
creation tools (e.g., elfweaver) or multiboot boot loaders can be
cumbersome, especially when many data files or shared libraries are
involved. To facilitate the grouping of files, 'tar_rom' is an
implementation of the 'ROM' interface that operates on a 'tar' file.
The name of the TAR archive must be specified via the 'name' attribute of
an 'archive' tag, for example:
! <config>
! <archive name="archive.tar"/>
! </config>
The backing store for the dataspaces exported via ROM sessions is accounted
on the 'rom_tar' service (not on its clients) to make the use of 'rom_tar'
transparent to the regular users of core's ROM service. Hence, this service
must not be used by multiple clients that do not trust each other.
Typically, 'tar_rom' is instantiated per client.
The Live CD uses the 'tar_rom' service for the browser demo. Each plugin
is fetched from the web as a tar file containing the config file of the
plugin subsystem as well as supplemental binary files that are provided
to the plugin subsystem as ROM files. This way, a plugin can carry along
multiple components and data that form a complete Genode subsystem.
DDE Kit
=======
The DDE kit underwent slight modifications since the previous release.
It now provides 64-bit integer types and a revised virtual PCI bus
implementation.
Device drivers
##############
PCI bus
=======
Genode was tested on several hardware platforms in preparation of the
current release. This revealed some deficiencies with the PCI bus
driver implementation. The revised driver now efficiently supports
platforms with many PCI busses (as PCIe demands) and correctly handles
multi-function devices.
VESA framebuffer
================
We updated the configuration syntax of the VESA driver to better match
the style of new init syntax, preferring the use of attributes rather than
XML sub nodes. Please refer to the updated documentation at
'os/src/drivers/framebuffer/vesa/README'.
:Buffered output:
To accommodate framebuffer clients that need to read from the frame buffer,
in particular the nitpicker GUI server operating with alpha channels, we
introduced a buffered mode to the VESA driver. If enabled, the VESA driver
will hand out a plain memory dataspace to the client rather than the
physical framebuffer. Each time, the client issues as 'refresh' operation
on the framebuffer-session interface, the VESA driver copies the corresponding
screen region from the client-side virtual framebuffer to the physical
framebuffer. Note that the VESA driver will require additional RAM quota
to allocate the client buffer. If the quota is insufficient, the driver will
fall back to non-buffered output.
:Preinitialized video modes:
As an alternative to letting the VESA driver set up a screen mode, the
driver has become able to reuse an already initialized mode, which is useful
if the VESA mode is already initialized by the boot loader. If the screen
is initialized that way, the 'preinit' attribute of the 'config' node can
be set to '"yes"' to prevent the driver from changing the mode. This way,
the driver will just query the current mode and make the already
initialized framebuffer available to its client.
Audio
=====
We observed certain hardware platforms (in particular VirtualBox) to
behave strangely after ALSA buffer-underrun conditions. It seems that the
VirtualBox audio driver plays actually more frames than requested by
ALSA's 'writei' function, resulting in recurring replay of data that
was in the buffer at underrun time. As a work-around for this problem,
we zero-out the sound-hardware buffer in the condition of an ALSA buffer
underrun. This way, the recurring replay is still there, but it is
replaying silence.
To improve the support for sporadic audio output, we added a check for the PCM
state for buffer underruns prior issuing the actual playback. In the event of
an underrun, we re-prepare the sound card before starting the playback.
Furthermore, we implemented the new flush and channel-synchronization
abilities of the 'Audio_out_session' interface for the DDE Linux driver.
Paravirtualized Linux
#####################
To support the demo scenarios that showcase the paravirtualized Linux kernel,
we enhanced our custom stub drivers of the OKLinux kernel. Thereby, we have
reached a high level of integration of OKLinux with native Genode services,
including audio output, block devices, framebuffer output, seamless integration
with the Nitpicker GUI, and networking. All stub drivers are compiled in by
default and are ready to use by specifying a device configuration in the config
node for the Linux kernel. This way, one Linux kernel image can be easily used
in different scenarios.
:Integration with the Nitpicker GUI:
We enhanced our fbdev stub driver with a mechanism to merge view reposition
events. If a X11 window is moved, a lot of subsequent events of this type are
generated. Using the new optimization, only the most recent state gets
reported to Nitpicker, making the X11 GUI more responsive.
:UnionFS:
As we noticed that unionfs is required by all our Linux scenarios, we decided
to include and enable the patch by default.
:Network support:
With the introduction of the 'nic_bridge', multiple networking stacks can run
on Genode at the same time, which paves the way for new use cases. We have now
added a stub driver using Genode's 'Nic_session' interface to make the new
facility available to Linux.
:Audio output:
We adapted the ALSA stub driver to the changes of the 'Audio_out_session'
interface, using the new channel synchronization and flush functions.
Thereby, we optimized the stub driver to keep latency and seek times of
Linux userland applications reasonably low.
:Removed ROM file driver:
With the addition of the 'Block_session' stub driver, the original ROM file
driver is no longer required. So we removed the stub. For using ROM files as
disk images for Linux, there is the 'rom_loopdev' server, which provides a
block session that operates on a ROM file.
:Asynchronous block interface:
To improve performance, we changed the block stub driver to facilitate the
asynchronous mode of operation as provided by the 'Block_session' interface.
This way, multiple block requests can be issued at once, thereby shadowing
the round trip times for individual requests.
Protocol stacks and libraries
#############################
Gallium3D / Intel GEM
=====================
We improved the cache handling of our DRM emulation code (implementing
'drm_clflush_pages') and our EGL driver, thereby fixing caching
artifacts on i945 GPUs. Furthermore, we added a temporary work-around
for the currently dysfunctional sequence-number tracking with i945 GPUs.
On this chipset, issuing the 'MI_STORE_DWORD_INDEX' GPU command used
for tracking sequence numbers apparently halts the processing the command
stream. This condition is normally handled by an interrupt. However,
we have not enabled interrupts yet.
To prepare the future support for more Gallium drivers than i915, we
implemented a driver-selection facility in the EGL driver. The code
scans the PCI bus for a supported GPU and returns the name of the
corresponding driver library. If no driver library could be found,
the EGL driver falls back to softpipe rendering.
lwIP
====
We revised our port of the lwIP TCP/IP stack, and thereby improved its
stability and performance.
* The lwIP library is now built as shared object, following the convention
for libraries contained in the 'libports' repository.
* By default (when using the 'libc_lwip_nic_dhcp' library), lwIP will
issue a DHCP request at startup. If this request times out, the loopback
device is set as default.
* If there is no 'Nic' service available, the lwIP stack will fall back to
the loopback device.
* We increased the default number of PCBs in lwIP to 64.
* We removed a corner case of the timed semaphore that could occur
when a timeout was triggered at the same time ,'up' was called.
In this case, the semaphore was unblocked but the timeout condition
was not reflected at the caller of 'down'. However, the lwIP code
relies on detecting those timeouts.
Qt4
====
We implemented a custom *nitpicker plugin widget*, which allows for the
seamless integration of arbitrary nitpicker clients into a Qt4 application.
The primary use case is the browser plugin mechanism presented at
the Live CD. In principle, the 'QNitpickerViewWidget' allows for creating
mash-up allocations consisting of multiple native Genode programs. As shown
by the browser plugin demo, a Qt4 application can even integrate other
programs that run isolated from the Qt4 application, and thereby depend on
on a significantly less complex trusted computing base than the Qt4
application itself.
[image nitpicker_plugin]
The image above illustrates the use of the 'QNitpickerViewWidget' in the
scenario presented on the Live CD. The browser obtains the Nitpicker view to be
embedded into the website from the loader service, which virtualizes the
Nitpicker session interface for the loaded plugin subsystem. The browser then
tells the loader about where to present the plugin view on screen. But it has
neither control over the plugin's execution nor can it observe any user
interaction with the plugin.
New Gems repository with HTTP-based block server
################################################
To give the web-browser demo of our Live CD a special twist, and to show off
the possibilities of a real multi-server OS, we decided to implement the
somewhat crazy idea of letting a Linux OS run on a disk image fetched at
runtime from a web server. This way, the Linux OS would start right away and
disk blocks would be streamed over the network as needed. Implementing this
idea was especially attractive because such a feature would be extremely hard
to implement on a classical OS but is a breeze to realize on Genode where all
device drivers and protocol stacks are running as distinct user-level
components. The following figure illustrates the idea:
[image http_block]
The block stub driver of the Linux kernel gets connected to a special block
driver called 'http_block', which does not access a real block device but
rather uses TCP/IP and HTTP to fetch disk blocks from a web server.
Because the 'http_block' server is both user of high-level functionality (the
lwIP stack) and provider of a low-level interface ('Block_session'), the
program does not fit well into one of the existing source-code repositories.
The 'os' repository, which is normally hosting servers for low-level interfaces
is the wrong place for 'http_block' because this program would make the 'os'
repository depend on the higher-level 'libports' repository where the 'lwip'
stack is located. On the other hand, placing 'http_block' into the 'libports'
repository is also wrong because the program is not a ported library. It merely
uses libraries provided by 'libports'. In the future, we expect that native
Genode components that use both low-level and high-level repositories will
become rather the norm than an exception. Therefore, we introduced a new
repository called 'gems' for hosting such programs.
Tools
#####
Automated coding-style checker
==============================
As Genode's code base grows and new developers start to get involved,
we noticed recurring questions regarding coding style. There is a
[https://genode.org/documentation/developer-resources/coding_style - document]
describing our coding style but for people just starting to get involved,
adhering all the rules can become tedious. However, we stress the importance
of a consistent coding style for the project. Not only does a consistent style
make the framework more approachable for users, but it also eases the work
of all regular developers, who can feel right at home at any part of
the code.
To avoid wasting precious developer time with coding-style fixes, we
have created a tool for the automated checking and (if possible) fixing
the adherence of source code to Genode's coding style. The tool is
located at 'tool/beautify'. It takes a source file as argument and
reports coding-style violations. The checks are fairly elaborative:
* Placement of braces and parenthesis
* Indentation and alignment, trailing spaces
* Vertical spacing (e.g., between member functions, above comments)
* Naming of member variables and functions (e.g., private members start with '_')
* Use of upper and lower case
* Presence of a file header with the mandatory fields
* Policy for function-header comments (comment at declaration, not
at implementation)
* Style of single-line comments, function-header comments, multi-line comments
The user of 'beautify' may opt to let the tool fix most of the violations
automatically by specifying the command line arguments '-fix' and '-write'.
With only the '-fix' argument, the tool will output the fixed version of
the code via stdout. By specifying the '-write' argument, the changes will
be written back to the original file. In any case, we strongly recommend
to manually inspect all changes made by the tool.
Under the hood, the tool consists of two parts. A custom C++ parser called
'parse_cxx' reads the source code and converts it to a syntax tree. In the
syntax tree, all formating information such as whitespaces are preserved.
The C++ parser is a separate command-line tool, which we also use for
other purposes (e.g., generating the API documentation at the website).
The actual 'beautify' tool calls 'parse_cxx', and applies its checks and
fixes to the output of 'parse_cxx'. For this reason, both tools have to
reside in the same directory.
Platform-specific changes
#########################
OKL4
====
:Added support for shared interrupts:
The Genode Live CD operates on a large number of devices that trigger
interrupts (USB, keyboard, mouse, ATAPI, timer, network). On most
platforms, the chances are extremely high that some of them use
the same IRQ line. Therefore, we enhanced core's IRQ service to
allow multiple clients to request the same IRQ. If the interrupt occurs,
all clients referring to this interrupt are notified. The interrupt
gets cleared after all of those clients responded. Even though, we regard
PIC interrupts as a legacy, the support of shared interrupts enables
us to use OKL4 with such complex usage scenarios.
:Revised page-fault handling:
If a page fault occurs, the OKL4 kernel delivers a message to the page-fault
handler. The message contains the page-fault address and type as well as the
space ID where the fault happened. However, the identity of the faulting
thread is not delivered. Instead, the sender ID of the page fault message
contains the KTCB index of the faulting thread, which is only meaningful
within the kernel. This KTCB index is used as a reply token for answering the
page fault message. We wondered about why OKL4 choose to deliver the KTCB
index rather then the global thread ID as done for plain IPC messages. The
only reasonable answer is that by using the KTCB index directly in OKL4's
page-fault protocol, one lookup from the userland-defined thread ID to the
KTCB index can be avoided. However, this comes at the cost of losing the
identity of the faulting thread. We used to take the space ID as a key for
the fault context within core. However, with Genode's user-level page-fault
mechanism, this simplification does not suffice anymore. We have to know the
faulting thread as a page fault may not be answered immediately but at a
later time. During that time, the page-fault state has to be stored at core's
representation of the faulting thread. Our solution is reverting OKL4's
page-fault protocol to operate with global thread IDs only and to never make
kernel-internal KTCB indices visible at the user land. You can find the patch
for the OKL4 kernel at 'base-okl4/patches/reply_tid.patch'.
:Reboot via kernel debugger:
We fixed the reboot code of OKL4's kernel debugger to improve our work
flow. The patch can be found at 'base-okl4/patches/kdb_reboot.patch'.
:Relieved conflict with libc 'limits.h':
For some reason, the OKL4 kernel bindings provide definitions
normally found in libc headers. This circumstance ultimately leads
to trouble when combining OKL4 with a real C runtime. We have
relieved the problem with the patch 'base-okl4/patches/char_bit.patch'.
:Exception handling:
We added a diagnostic message to core that reports about exceptions
such as division by zero.
Pistachio
=========
Our revised syscall bindings for supporting position-independent code
on L4ka::Pistachio have been integrated into the mainline development
of the kernel. Therefore, the patch is not needed anymore when using
a kernel revision newer than 'r791:0d25c1f65a3a'.
Linux
=====
On Linux, we let the kernel manage all virtual address spaces for us,
except for the thread-context area. Because the kernel does not know
about the special meaning of the thread-context area, it may choose to
use this part of the virtual address space as target for 'mmap'. This
may lead to memory corruption. Fortunately, there is a way to tell the
kernel about virtual address regions that should be reserved. The
trick is to pre-populate the said region with anonymous memory using
the 'mmap' arguments 'MAP_PRIVATE', 'MAP_FIXED', 'MAP_ANONYMOUS', and
'PROT_NONE'. The kernel will still accept a fixed-address mapping
within such a reserved region (overmap) but won't consider using the
region by itself. The reservation must be done at the startup of each
process and each time when detaching a dataspace from the thread
context area. For the process startup, we use the hook function
'main_thread_bootstrap' in 'src/platform/_main_helper.h'. For reverting
detached dataspaces to a reserved region within the context area, we
added as special case to 'src/base/env/rm_session_mmap.cc'.
For hybrid programs (Genode processes that link against native
shared libraries of the Linux system), which are loaded by the dynamic
linker of Linux, we must further prevent the dynamic linker from
populating the thread-context area. This is achieved by adding a
special program segment at the linking stage of all elf binaries.

876
doc/release_notes/11-02.txt Normal file
View File

@@ -0,0 +1,876 @@
===============================================
Release notes for the Genode OS Framework 11.02
===============================================
Genode Labs
One year ago, the release 10.02 was our break-through with regard to the support
of multiple kernels as base platform for Genode. With the added support for
the NOVA hypervisor and the Codezero kernel, Genode applications could be executed
on 6 different kernels. With the current release, we take our commitment to
kernel platform support even further. With the added support for the Fiasco.OC
kernel, we make Genode available on one of the most feature-rich modern microkernels.
Additionally, we entered the realms of kernel design with our new platform support
for the Xilinx MicroBlaze architecture. This platform support comes in the shape
of a custom kernel specifically targeted to the MicroBlaze CPU architecture.
Furthermore, we updated our support for the NOVA Hypervisor to the bleeding-edge
version 0.3, which has been released earlier this month.
With the current support for 8 different kernel platforms (L4/Fiasco, Linux,
L4ka::Pistachio, OKL4, NOVA, Codezero, Fiasco.OC, and native MicroBlaze), testing
and integrating application scenarios across all platforms becomes increasingly
challenging. Therefore, we introduce a new framework for automating such tasks.
Thanks to the tight integration of the automation tool with Genode's build system,
going back and forth between different kernels becomes an almost seamless
experience.
Functionality-wise, the release carries on our vision to create a highly secure
yet easy to use general-purpose operating system. Because the Genode framework
is developed on Linux using the wonderful GNU tools, we consider the
availability of the GNU user land on Genode as crucial for using the system by
ourself. This motivation drives the creation of a custom execution environment
for GNU software on top of Genode. With the current release, we are proud to
present the first pieces of this execution environment. Even though not fully
functional yet, it clearly shows the direction of where we are heading.
Support for Fiasco.OC
#####################
The OC in the name of the Fiasco.OC kernel stands for "object capability", hinting
at the most significant feature that sets current-generation microkernels such as
NOVA, seL4, and Fiasco.OC apart from their predecessors. Whereas previous L4 kernels
succeeded in protecting subsystems from each other, the new generation of kernels
is geared towards strict security policies. Traditionally, two protection domains
were able to communicate with each other if they both agreed. Communication partners
were typically globally known via their respective thread/task IDs. Obviously, this
policy is not able to guarantee the separation of subsystems. If two subsystems
conspire, they could always share information. Object-capability-based kernels
are taking the separation much further by prohibiting any communication between
protection domains by default. Two protection domains can communicate only if
a common acquaintance of both agrees. This default-deny policy facilitates the
creation of least-privilege security policies. From the ground up, Genode has
been designed as a capability-based system which is naturally capable of leveraging
kernel-based object-capability support if present. After NOVA, Fiasc.OC is the
second of Genode's base platforms that provides this feature.
Apart from being a capability-based kernel, Fiasco.OC has a number of compelling
features such as thorough support for ARM platforms and the x86 32/64 bit
architectures. It supports SMP, hardware virtualization, and provides special
optimizations for running paravirtualized operating systems.
Technically, Fiasco.OC is the successor of the L4/Fiasco kernel developed by
the OS group of the TU-Dresden. However, the kernel interface of Fiasco.OC has
not much in common with L4/Fiasco. Some heritages are still there (e.g., IPC
timeouts) but the kernel API has evolved to a fully object-oriented model.
:Thanks:
We are indebted to the main developer of Fiasco.OC Alexander Warg for being
very reponsive to our inquiries while doing the porting work. Thanks to his
support, the adaptation of Genode to this kernel has been an almost smooth
ride.
Prerequisites
=============
You need GNU C & C++ Compilers, GNU Binutils, GNU Make, and Perl to use the
Fiasco.OC build system. On Debian/Ubuntu systems, you have to install the
following packages:
! apt-get install make gawk g++ binutils pkg-config subversion
Moreover, you need to download and install the tool-chain used by Genode. Have
a look at this page:
:[https://genode.org/download/tool-chain]:
Genode tool-chain
Downloading and building Fiasco.OC
==================================
Checkout the Fiasco.OC sources and tool-chain to an appropriated directory:
! export REPOMGR_SVN_REV=27
! svn cat https://svn.tudos.org/repos/oc/tudos/trunk/repomgr |\
! perl - init https://svn.tudos.org/repos/oc/tudos fiasco l4re
Building the kernel
~~~~~~~~~~~~~~~~~~~
Create the build directory for the kernel:
! cd <path_to_fiasco_src_dir>/src/kernel/fiasco
! make BUILDDIR=<path_to_kernel_build_dir>
Go to the build directory, configure the kernel:
! cd mybuild
! make config
This will launch the configuration menu. Here you can configure your kernel.
The default config is just fine to test the Genode port. It will build a
uniprocessor IA32 kernel with debugging features enabled. You can exit the menu and
save the configuration by simply typing 'x'.
Now, build Fiasco.OC by invoking:
! make
Building necessary tools
~~~~~~~~~~~~~~~~~~~~~~~~
To practically use Fiasco.OC, you need in addition to the kernel a tool to
bootstrap it, and the initial pager of the system, namely 'sigma0'. Both tools
can be found in the L4 runtime environment's base directory. Outgoing from
the directory where you checked out the sources, you have to change to the
following directory:
! cd <path_to_fiasco_src_dir>/src/l4
Create another build directory:
! make B=<path_to_l4re_build_dir>
Again, you might want to tweak the configuration:
! make O=<path_to_l4re_build_dir> config
Finally, build the tools:
! make O=<path_to_l4re_build_dir>
Building the Fiasco.OC version of Genode
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The Fiasco.OC version of Genode is available at the Genode public subversion repository:
:https://genode.org/download/subversion-repository:
Information about accessing the Genode public subversion repository
Go to a directory where you want the Genode/Fiasco.OC build directory to remain. Use
the helper script in the 'tool/builddir' directory of the Genode source tree to
create the initial build environment. You need to state the absolute path to the
build directory of the L4 runtime environment as 'L4_DIR', as it contains the kernel
bindings needed by the Genode port.
! <path_to_genode_src_dir>/tool/builddir/create_builddir foc_x86_32 \
! L4_DIR=<path_to_l4re_build_dir> \
! GENODE_DIR=<path_to_genode_src_dir> \
! BUILD_DIR=<path_to_genode_build_dir>
Now, go to the newly created build directory and type make.
! cd <path_to_genode_build_dir>
! make
Booting Genode on top of Fiasco.OC
==================================
Example GRUB configuration entry:
! timeout 0
! default 0
!
! title Genode on Fiasco.OC
! kernel /bootstrap -modaddr=0x01100000
! module /fiasco -serial_esc
! module /sigma0
! module /core
! module /init
! module /config
! module /pci_drv
! module /vesa_drv
! module /ps2_drv
! module /timer
! module /nitpicker
! module /launchpad
! module /liquid_fb
! module /scout
! module /testnit
! module /nitlog
For an example of a matching Genode 'config' file, please take a look
at 'os/config/demo'.
The Genode binaries are located in '<path_to_genode_build_dir>/bin',
the 'fiasco' kernel in '<path_to_kernel_build_dir>'. Assuming you compiled
for x86/586 (the default), you can find the 'bootstrap' binary in
'bin/x86_586' and 'sigma0' in 'bin/x86_586/l4f' within the
'<path_to_l4re_build_dir>' directory.
Current state
=============
The adaptation of Genode to Fiasco.OC covers most parts of the Genode API
including advanced semantics such as cancelable locks and support for
real-time priorities. So far, it has been tested on the x86 architecture.
Because 'base-foc' does not contain x86-specific code, we expect no major
roadblocks for running Genode on Fiasco.OC on ARM. However, we have not
exercised tests in this regard.
As of today, there exist the following limitations of the Fiasco.OC support:
* The dynamic linker is not yet adapted to Fiasco.OC. Special care must
be taken for handling the parent capability for dynamically loaded
programs. We have already covered this issue for the NOVA version but
the adaptation to Fiasco.OC remains yet to be done.
* The destruction of sub systems is not yet fully stable. Because Genode
forms a more dynamic workload than the original userland accompanied with
the kernel, the usage pattern of the kernel API triggers different
effects. We are working with the Fiasco.OC developers to remedy this
issue.
* The signalling framework is not yet supported. A design exist but it is
not implemented yet.
We believe however that none of these limitations are a significant hurdle for
starting to use Genode with this kernel. Please expect this issues to be
resolved with the upcoming Genode release.
Technical details about 'base-foc'
==================================
The following technical bits are worth noting when exploring the use of
Genode with the 'base-foc' platform.
* The timer implementation uses a one thread-per-client mode of operation.
We use IPC timeouts as time source. Hence, the timer driver is hardware
independent and should work out of the box on all hardware platforms
supported by Fiasco.OC.
* Each 'Server_object' of Genode corresponds to a so-called IPC gate,
which is the Fiasco.OC kernel object used for capability invocation.
Therefore, protection and object integrity is provided at the fine
granularity of single 'Server_objects'. This is in line with our
support for NOVA's implementation of capability-based security.
* In contrast to the lock implementation that we used with the original
L4/Fiasco kernel, the 'base-foc' lock is a fully-featured Genode lock
with support for lock cancellation and blocking. For blocking and
waking up lock applicants, we use Fiasco.OC's IRQ objects.
* The allocator used for managing process-local capability selectors
does not yet support the reuse of capability selectors.
Further Information
===================
:genode/tool/builddir/README:
Reference manual for the 'create_builddir' script
:[https://os.inf.tu-dresden.de/fiasco]:
Official website for the Fiasco.OC microkernel.
Noux - an execution environment for the GNU userland
####################################################
Even though Genode is currently mainly geared to the classical special-purpose
application domains for microkernel-based systems, the main property that sets
Genode apart from traditional systems is the thorough support for dynamic
workloads and the powerful mechanisms for handling hardware resources and
security policies in highly dynamic setting. We are convinced that Genode's
architecture scales far beyond static special-purpose domains and believe in
the feasibility of Genode as a solid foundation for a fully-fledged general
purpose operating system. Internally at Genode Labs, we set up the ultimate
goal to switch from Linux to Genode for our day-to-day work. We identified
several functionalities that we could not live without and systematically try
to bring those features to Genode. Of course, the most fundamental programs
are the tools needed to develop and build Genode. Currently we are developing
on Linux and enjoy using the GNU userland.
Consequently, we require a solution for using this rich tool set on Genode.
The straight-forward way for making these tools available on Genode would be
running them within a virtualized Linux instance (e.g., using OKLinux on OKL4).
However, this approach would defeat our actual goal to create a highly secure
yet easy to use working environment because adding Linux to the picture would
involve administering the virtualized Linux system. We would prefer a native
solution that makes the overall system less, not more, complicated. This way
the idea for a native execution environment for the GNU userland on Genode
was born. The implementation is called Noux and the first bits of code are
featured in the 'ports' repository. Noux consists of two parts, a build
environment for compiling GNU programs such that they can be run as Genode
processes and an execution environment that provides the classical UNIX
functionality to these programs.
Noux build environment
======================
From our experience, porting existing UNIX applications to a non-UNIX system
tends to be a task of manual and time-consuming labour. One has to loosely
understand the build system and the relationship of the involved source codes,
implement dummy functions for unresolved references, and develop custom glue
code that interfaces the ported application to the actual system. Taking the
shortcut of changing the original code has to be avoided at any cost because
this produces recurring costs in the future when updating the application. In
short, this long-winding process does not scale. For porting a tool set such as
the GNU userland consisting of far more than a three-digit number of individual
programs, this manual approach becomes unfeasible. Therefore, we have created
a build environment that facilitates the use of the original procedure of
invoking './configure && make'. The challenge is to supply configure with
the right arguments and environment variables ('CFLAGS' and the like) such that
the package is configured against the Genode environment. The following
considerations must be taken:
* Configure must not detect any global headers (e.g., '/usr/include/')
or libraries (e.g., '/usr/lib/'). This can be achieved by the '-nostdinc' and
'-nostdlib' options
* Configure has to use the same include-search paths as used for compiling
normal libc-using Genode programs
* Configure must use the Genode tool chain
* The final linking stage must use the Genode linker script, the Genode
base libraries, and other Genode-specific linker arguments.
Thanks to the power of the GNU build system, all this can be achieved by
supplying arguments to './configure' and indirectly to the 'make' process via
environment variables. The new Noux build environment takes care of these
precautions. It comes in the form of the 'ports/mk/noux.mk' file which enables
the seamless integration of GNU packages with the Genode build system. To
compile a GNU package, the manual steps needed are reduced to the creation of a
'target.mk' file representing the package. This 'target.mk' defines the name
of the package (by default, the basename of the 'target.mk' enclosing directory
is assumed) and the location of the source package. With this approach, we
managed to build 'coreutils' (over 100 small UNIX utilities such as 'ls', 'cp',
'sort'), 'binutils' (GNU linker, assembler, object-file tools), 'findutils'
('find', 'xargs'), 'bash', 'dash', GNU make, and finally the GNU compiler
collection including 'g++'. The resulting binaries are ready to be executed as
native Genode processes. However, without the right environment that presents
the program the needed UNIX functionality, those programs won't do much.
This leads us to the Noux execution environment.
Noux execution environment
==========================
The Noux execution environment plays the role of a UNIX kernel for programs
built via the Noux build environment. In contrast to a real kernel, the Noux
environment is a plain Genode user-level process that plays the role of being
the parent of one or multiple Noux processes. In addition of providing the
'Genode::Parent' interface, Noux also provides a locally implemented service called
'Noux::Session' that offers UNIX-like system-calls via an RPC interface. Each
hosted program is linked against a special Noux libc plugin that catches all
libc calls that would normally result in a system call. It then transparently
forwards this function call to the 'Noux::Session' interface.
Currently the Noux execution environment implements the following
system calls: 'getcwd', 'write', 'stat', 'fstat', 'fcntl', 'open',
'close', 'dirent', 'fchdir', 'read', and 'execve'.
The execution environment submits arguments (argc, argv, environment) to the
hosted program, manages its current working directory and receives its exit
code. File operations are targeted to a custom VFS infrastructure, which
principally allows a flexible configuration of the virtual file system visible
to the hosted programs. At the current stage, Noux supports mounting plain tar
archives obtained from core's ROM service as read-only file system. On startup,
the Noux environment starts one process (the init process) and connects the
file descriptor 1 (stdout) to Genode's LOG service.
State of the implementation
~~~~~~~~~~~~~~~~~~~~~~~~~~~
The infrastructure implemented so far already allows the execution of many simple
UNIX tools such as 'ls -lRa', 'echo', 'seq', 'find'. The 'execve' system call
is implemented such that a new process is started that inherits the file
descriptors and the PID of the calling process. This allows using the exec
functionality of the 'bash' shell. However, because 'fork' is not implemented
yet, there is currently no way to start multiple programs hosted in a single
Noux execution environment.
As of today, the Noux environment is not considered to be usable for practical
purposes. However, it clearly shows the feasibility of the path we are walking.
With the foundation laid, we are looking forward to expanding Noux to a capable
solution for running our beloved GNU userland tools on Genode.
Vision
~~~~~~
The most significant intermediate result of pursuing the development of Noux is
the realization that such an environment is not exceedingly complex. Because of
the combination with Genode, we only need to provide a comfortable runtime as
expected by user processes but we can leave much of intricate parts of UNIX out
of the picture. For example, because we handle device drivers on Genode, we do
not need to consider device-user interaction in Noux. As another example,
because the problem of bootstrapping the OS is already solved by Genode, there
is no need to run an 'init' process within Noux. Our vision foresees that Noux
runtimes are to be created on demand for individual tasks such as editing a
file (starting a custom Noux instance containing only the file to edit and the
text editor), compiling source code (starting a custom Noux instance with only
the source code and the build tools). Because Noux is so simple, we expect the
runtime overhead of starting a Noux instance to be not more than the time
needed to spawn a shell in a normal UNIX-like system.
Test drive
~~~~~~~~~~
To give Noux a spin, we recommend using Linux as base platform as this is
the platform we use for developing it. First, you will need to download the
source code of the GNU packages. From within the 'ports' repository,
use the following command:
! make prepare PKG=coreutils
This command will download the source code of the GNU coreutils. You may
also like to give the other packages a try. To see what is available,
just call 'make' without any argument.
Create a build directory (e.g., using tool/builddir/create_builddir).
Change to the build directory and issue the command
! make run/noux
This command will execute the run script provided at 'ports/run/noux.run'.
First it builds core, init, and coreutils. Then it creates a tar archive
containing the installed coreutils. Finally, it starts the Noux environment on
Genode. Noux then mounts the TAR archive as file system and executes 'ls -laR',
showing the directory tree.
Approaching platform support for Xilinx MicroBlaze
##################################################
With the release 11.02, we are excited to include the first version of our
custom platform support for the Xilinx MicroBlaze CPU architecture. MicroBlaze
is a so-called softcore CPU, which is commonly used as part of FPGA-based
System-on-Chip designs. At Genode Labs, we are regularly using this IP core,
in particular for our Genode FPGA Graphics Project, which is a GUI software stack
and a set of IP cores for implementing fully-fledged windowed GUIs on FPGAs:
:Website of the Genode FPGA Graphics Project:
[https://genode-labs.com/products/fpga-graphics]
Ever since we first released the Genode FPGA project, we envisioned to combine
it with the Genode OS Framework. In Spring 2010, Martin Stein joined our team
at Genode Labs and accepted the challenge to bring the Genode OS Framework to
the realms of FPGA-based SoCs. Technically, this implies porting the framework
to the MicroBlaze CPU architecture. In contrast to most softcore CPUs such as
the popular Lattice Mico32, the MicroBlaze features a MMU, which is a fundamental
requirement for implementing a microkernel-based system. Architecturally-wise
MicroBlaze is a RISC CPU similar to MIPS. Many system parameters of the CPU
(caches, certain arithmetic and shift instructions) can be parametrized at
synthesizing time of the SoC. We found that the relatively simple architecture
of this CPU provides a perfect playground for pursuing some of our ideas about
kernel design that go beyond the scope of current microkernels. So instead of
adding MicroBlaze support into one of the existing microkernels already
supported by Genode, we went for a new kernel design. Deviating from the typical
microkernel, which is a self-sufficient program running in kernel mode that
executes user-level processes on top, our design regards the kernel as a part of
Genode's core. It is not a separate program but a library that implements the
glue between user-level core and the raw CPU. Specifically, it provides the
entrypoint for hardware exceptions, a thread scheduler, an IPC mechanism, and
functions to manipulate virtual address spaces (loading and flushing entries
from the CPU's software-loaded TLB). It does not manage any physical memory
resources or the relationship between processes. This is the job of core.
From the kernel-developer's point of view, the kernel part can be summarized as
follows:
* The kernel provides user-level threads that are scheduled in a round-robin
fashion.
* Threads can communicate via synchronous IPC.
* There is a mechanism for blocking and waking up threads. This mechanism
can be used by Genode to implement locking as well as asynchronous
inter-process communication.
* There is a single kernel thread, which never blocks in the kernel code paths.
So the kernel acts as a state machine. Naturally, there is no concurrency in the
execution paths traversed in kernel mode, vastly simplifying these code parts.
However, all code paths are extremely short and bounded with regard to
execution time. Hence, we expect the interference with interrupt latencies
to be low.
* The IPC operation transfers payload between UTCBs only. Each thread has a
so-called user-level thread control block which is mapped transparently by
the kernel. Because of this mapping, user-level page faults cannot occur
during IPC transfers.
* There is no mapping database. Virtual address spaces are manipulated by
loading and flushing physical TLB entries. There is no caching of mappings
done in the kernel. All higher-level information about the interrelationship
of memory and processes is managed by the user-level core.
* Core runs in user mode, mapped 1-to-1 from the physical address space
except for its virtual thread-context area.
* The kernel paths are executed in physical address space (MicroBlaze).
Because both kernel code and user-level core code are observing the same
address-space layout, both worlds appear to run within a single address
space.
* User processes can use the entire virtual address space (4G) except for a
helper page for invoking syscalls and a page containing atomic operations.
There is no reservation used for the kernel.
* The MicroBlaze architecture lacks an atomic compare-and-swap instruction. On
user-level, this functionality is emulated via delayed preemption. A kernel-
provided page holds the sequence of operations to be executed atomically and
prevents (actually delays) the preemption of a thread that is currently
executing instructions at that page.
* The MicroBlaze MMU supports several different page sizes (1K up to 16MB).
Genode fully supports this feature for page sizes >= 4K. This way, the TLB
footprint can be minimized by choosing sensible alignments of memory
objects.
Current state
=============
The MicroBlaze platform support resides in the 'base-mb' repository. At the
current stage, core is able to successfully start multiple nested instances of
the init process. Most of the critical kernel functionality is working. This
includes inter-process communication, address-space creation, multi-threading,
thread synchronization, page-fault handling, and TLB eviction.
This simple scenario already illustrates the vast advantage of
using different page sizes supported by the MicroBlaze CPU. If using
4KB pages only, a scenario with three nested init processes produces more than
300.000 page faults. There is an extremely high pressure on the TLB, which
only contains 64 entries. Those entries are constantly evicted so that
threshing effects are likely to occur. By making use of flexible page
sizes (4K, 16K, 64K, 256K, 1M, 4M, 16M), the number of page faults gets
slashed to only 1.800, speeding up the boot time by factor 10.
Currently, there is no restriction of IPC communication rights. Threads are
addressed using their global thread IDs (in fact, using their respective
indices in the KTCB array). For the future, we are planning to add
capabilty-based delegation of communication rights.
Building and using Genode on MicroBlaze
=======================================
For building Genode for the MicroBlaze platform, you need the MicroBlaze
tool chain as it comes with the Xilinx EDK. The tool chain is typically
prefixed with 'mb-'. Please make sure that the tool chain's 'bin/' directory
is included in your 'PATH' environment variable.
For building and starting Genode on MicroBlaze, you first need to create
a build directory using the build-directory creation tool:
! tool/builddir/create_builddir microblaze \
! BUILD_DIR=</path/to/build/dir> \
! GENODE_DIR=</path/to/genode/dir>
The 'base-mb' repository comes with support for Genode's run tool. In order to
use it, you will first need to declare the location of your qemu binary using
the 'QEMU=/path/to/qemu' variable in the '<build-dir>/etc/microblaze.conf'
file. Then you will be able to start an example scenario by issuing the
following command from within your build directory:
! make run/nested_init
Thereby, the 'run' tool will attempt to start core using the microblaze version
of qemu.
You can also find a simple hello-world example at 'base-mb/src/test/hello'.
The corresponding run script is located at 'base-mb/run/hello.run'. You can
execute it via 'make run/hello' from the build directory.
Note that currently, all boot modules are linked against the core binary.
To change the boot modules, the file 'base-mb/src/core/boot_modules.s' must
be modified.
For reference, we are using the following tools:
* mb-g++ (GCC) 4.1.1 20060524 (Xilinx 11.2 Build EDK_LS2.2
20 Apr 2009 Xilinx 11.2 Build EDK_LS2.2 23 Apr 2009)
* GNU ld version 2.16 Xilinx 11.2 Build EDK_LS2.2 23 Apr 2009
* GNU assembler 2.16 Xilinx 11.2 Build EDK_LS2.2 23 Apr 2009
* QEMU emulator version 0.14.50, Copyright (c) 2003-2008 Fabrice Bellard
Petalogix linux reference design targeting Xilinx Spartan 3ADSP-1800 boards.
Supporting the NOVA hypervisor version 0.3
##########################################
NOVA is a so called microhypervisor - a modern capability-based microkernel
with special support for hardware-based virtualization and IOMMUs. Since we
incorporated the initial support for the NOVA hypervisor in Genode one year
ago, this kernel underwent multiple revisions. The latest version was released
earlier this month. To our delight, much of the features that we missed from
the initial release had been implemented during the course of the last year. We
are especially happy about the fully functional 'revoke' system call and the
support for remote kernel-object creation.
With the Genode release 11.02, we officially support the latest NOVA version.
The update of Genode to the new version required two steps. First, because many
details of the kernel interface were changed between version 0.1 and version
0.3, we had to revisit our syscall bindings and adapting our code to changed
kernel semantics. Second, we filled our 'base-nova' code related to object
destruction and unmapping with life to benefit from NOVA's 'revoke' system
call. Consequently, we are now able to run the complete Genode software stack
including the dynamic linker on NOVA.
Note that for using Genode on NOVA, you will need to apply a small patch to the
NOVA source code. This patch enables the re-use of user-level thread control
blocks in the kernel. The patch can be found at 'base-nova/patches/utcb.patch'.
When executing NOVA on qemu, please specify the '-cpu coreduo' argument to the
qemu command line. When using Genode 'run' tool, you may assign this argument
to the 'QEMU_OPT' variable in '<build-dir>/etc/build.conf'.
:Thanks:
We are grateful for the ongoing very pleasant collaboration with Udo Steinberg
who is the driving force behind NOVA. Thanks for the ultra-fast responses to our
questions and for considering our suggestions regarding the feature set of
NOVA's kernel interface!
Base framework
##############
Upgrading existing sessions
===========================
Genode enables a client of a service to lend parts of its own resources to
the service when opening a session. This way, servers do not need to allocate
own resources on behalf of their clients and become inherently robust against
resource-exhaustion-based denial-of-service attacks.
However, there are cases when the client can not decide about the amount of
resources to lend at session-creation time. In such cases, we used to devise an
overly generous client policy. Now, we have added a new 'upgrade' function to
the 'Parent' and 'Root' interfaces that enables a client to upgrade the
resources of an existing session.
For the 'env()->rm_session()' and 'env()->ram_session()' of processes using
the Genode 'env' library, we implemented a transparent quota upgrade that kicks in
in the event of an exceeded metadata backing store.
Comprehensive accounting of core resources
==========================================
We changed all services of core to limit their respective resource usage
specifically for each individual session. For example, the number of dataspaces
that can be handled by a particular region-manager (RM) session depends on the
resource donation attached to the session. To implement this accounting scheme
throughout core, we added a generic 'Allocator_guard' utility to
'base/include/'. We recommend using this utility when implementing resource
multiplexers, in particular multi-level services. Thanks to this change in
core, the need for a slack memory reservation in core has vanished.
Various changes
===============
The remaining parts of the base API underwent no fundamental revision. The
changes are summarized as follows.
:C++ Support:
We removed 'libgcc' from our C++ support library ('cxx') and link
it to each individual final target and shared library instead. This change alleviates
the need to abuse the 'KEEP_SYMBOLS' mechanism that we used in 'cxx' to
keep libc-dependencies of GCC's support libraries local to the 'cxx'
library. Besides the benefit of reducing heuristics, this change improves
the compatibility with recent cross-compiling tool chains.
Furthermore, we added 'realloc' to the local libc support of the 'cxx'
library because recent ARM tool chains tend to use this function.
:Argument handling for 'main()':
We added a hook to the startup code to enable the implementation of
custom facilities for passing arguments to the main function. The
hook uses the global variables 'genode_argc' and 'genode_argv'.
:Child-exit policy hook:
We enhanced the 'Child_policy' with a new policy interface that allows
a simplified implementation of policies related to program termination.
:Changed API of 'Range_allocator':
We changed the return value of 'alloc_addr' to distinguish different error
conditions. Note that the boolean meaning of the return value is inverted.
Please check your uses of 'alloc_addr'!
Operating-system services and libraries
#######################################
C Runtime
=========
In conjunction with our work on Noux, we improved Genode's C runtime at many
places. First, we added libstdtime and some previously missing bits of libgdtoa
to the libc. These additions largely alleviate the need for dummy stubs, in
particular time-related functions. Second, we added the following functions to
our libc plugin interface: 'dup2', 'fchdir', 'fcntl', 'fstat', 'stat', and
'write'. This enables the creation of advanced libc plugins simulating a whole
file system as done with Noux. Still, there are a number of dummy stubs found
at 'libc/src/lib/libc/dummy.cc'. However, those stubs are now all defined as
weak symbols such that they can be overridden by libc plugins. Finally, we have
replaced the original 'exit' implementation that comes with the libc with a
Genode-specific version. The new version reports the exit code of the
application to the parent process via an 'Parent::exit()' RPC call.
Until now, Genode's libc magically handled output to stdout and stderr by
printing messages via Genode's LOG interface. We have now replaced this
hard-wired interface by an optional libc plugin called 'libc_log'. If present, write
operations to stdout are caught at the libc plugin interface and delegated to
the plugin, which implements the output to the LOG interface. If you have an
application using Genode's libc, you might consider adding the 'libc_log'
library to your 'target.mk' file.
Support for big numbers by the means of libgmp and libmpfr
==========================================================
We have now include both the GNU Multiple Precision Arithmetic Library and
(GMP) and MPFR to the 'ports' repository. This work was specifically motivated
by our port of GCC to Genode as GCC version 4.4.5 requires both libraries.
Because we intend to use those libraries primarily on x86_32, the current port
covers only this architecture. However, expanding the port to
further CPU architectures should be straight-forward if needed.
Furthermore, you can now also find GCC's 'longlong.h' header at
'libports/include/gcc'.
Qt4 updated to version 4.7.1
############################
The current release bumps the supported Qt4 version from 4.6.2 to 4.7.1 and the
Arora web browser (located at the ports repository) from version 0.10.2 to
version 0.11. Of course, we updated our custom additions such as our custom
Nitpicker plugin widget that enables the seamless integration of native
Nitpicker GUI clients into Qt4 applications to work with the new Qt4 version.
Tools
#####
Tool chain update to GCC 4.4.5 and Binutils 2.21
================================================
We upgraded the official Genode tool chain from gcc 4.2.4 to gcc 4.4.5. Please
update your tool chain by downloading the new binary archive (available for x86_32)
or building the tool chain from source using our 'tool/tool_chain' utility.
New support for automated integration and testing
=================================================
With the growing number of supported base platforms, the integration and testing
of Genode application scenarios across all kernels becomes
increasingly challenging. Each kernel has a different boot mechanism and
specific requirements such as the module order of multiboot modules (Fiasco's
bootstrap, Pistachio's sigma0 and kickstart), kernel parameters, or the
invocation of a single-image creation tool (OKL4's elfweaver). To make our
life supporting all those platforms easier, we have created a tool called
'run', which is tightly integrated within Genode's build system. In short 'run'
gathers the intrinsics in the form of a 'run/env' file specific for the
platform used by the current build directory from the respective
'base-<platform>' repository. It then executes a so-called run script, which
contains all steps needed to configure, build, and integrate an application
scenario. For example, a typical run script for building and running a test
case resides in a file called '<any-repository>/run/<run-script-name>.run' and
looks as follows:
! build "core init test/exception"
! create_boot_directory
! install_config {
! <config>
! <parent-provides>
! <!--<service name="ROM"/>-->
! <service name="LOG"/>
! </parent-provides>
! <default-route>
! <any-service> <parent/> </any-service>
! </default-route>
! <start name="test-exception">
! <resource name="RAM" quantum="1M"/>
! </start>
! </config>
! }
! build_boot_image "core init test-exception"
! append qemu_args "-nographic -m 64"
! run_genode_until {.*Exception \(label 0xffb0\) occured.*} 10
First, the build system is instructed to create the targets specified as argument
for the 'build' function. Next, for the integration part, a new boot directory is
created. On most kernel platform, the respective location of the boot directory
is '<build-dir>/var/run/<run-script-name>'. Initially, this directory is empty.
It gets populated with a 'config' file specified as argument of the 'install_config'
command, and by the boot modules specified at the 'build_boot_image' command.
Now that the integration is complete, the scenario is executed via the
'run_genode_until' command. This command takes a regular expression as
argument, which determines the successful termination of the test case. The
second argument is a timeout (is seconds). In the example, the test case will
fail if its output does not match the regular expression within the execution
time of 10 seconds.
The command 'append qemu_args' specifies run-script-specific qemu arguments in
the case that qemu is used to execute the scenario. This is the case for most
kernel platforms (except for Linux where core gets executed directly on the host).
Additional build-directory-specific qemu arguments can be specified in the
'etc/build.conf' file by defining the 'QEMU_OPT' variable. For example, to
prevent KVM being used on Ubuntu Linux, specify:
! QEMU_OPT = -no-kvm
To execute the run script from with build directory, you need to have Expect
installed. Typically, the Linux package is called 'expect'. Simply issue
the following command from within your build directory:
! make run/<run-script>
Note that you will need to have a GRUB 'stage2_eltorito' binary available
at '<genode-dir>/tool/grub' on base platforms that use an ISO image as boot
stategy.
Because the whole chain of actions, building, integrating, executing, and
validating an application scenario is now at the fingertips of issuing a
single command with no kernel-specific considerations needed, it has never
been easier to run the same scenario on a wide range of different kernels.
Please find further instructive examples at 'os/run/'. The 'ldso' run
script executes the test of the dynamic linker. It is completely generic.
The 'demo' run script starts Genode's default demo scenario and shows how
platform-specific considerations (e.g., which device drivers to use) can be
taken into account.
We found that the 'run' tool significantly boosted our productivity not
only for testing purposes but also for accelerating the development-test
cycle during our day-to-day work.
:Technical notes:
The 'run' tool uses Expect as automation tool. Expect is a Tcl interpreter,
which is accompanied by special functionality for automating interactive
command-line applications. Technically, a run script is an Expect script
which gets included by the 'tool/run' script. For the reference of
run-specific functions, please revise the documentation in the 'tool/run'
script. Because each run script is actual Expect source code, it is possible
to use all Tcl and Expect scripting features in a run script.
In particular, a run script may issue shell commands using Tcl's 'exec'
function. This way, even complex integration tasks can be accomplished.
For example, the integration of the Genode Live CD was done via a single
run script.
Build system
============
To facilitate the integration of 3rd-party build systems into the Genode build
process, we added support for pseudo targets that do not require any 'SRC'
declaration. Such 'target.mk' may contain custom rules that will be executed
when the target is revisited by the build system. The bindings are as follows:
! build_3rd_party:
! ...custom commands...
!
! $(TARGET): build_3rd_party
!
! clean_3rd_party:
! ...custom commands...
!
! clean_prg_objects: clean_3rd_party:

1289
doc/release_notes/11-05.txt Normal file
View File

File diff suppressed because it is too large Load Diff

703
doc/release_notes/11-08.txt Normal file
View File

@@ -0,0 +1,703 @@
===============================================
Release notes for the Genode OS Framework 11.08
===============================================
Genode Labs
One of Genode's most distinctive properties is its support for various
different kernels as base platforms. Each of the 8 currently supported kernels
differs with regard to features, security, hardware support, complexity, and
resource management. Even though different applications call for different
kernel properties, through Genode, those properties can be leveraged using a
unified API. The growing number of supported base platforms, however, poses two
challenges, which are the comprehension of the large diversity of tools and
boot concepts, and capturing of the semantic differences of all the kernels.
With the version 11.08, the framework mitigates the former challenge by
introducing a unified way to download, build, and use each of the
kernels with Genode's user-level infrastructure. The new tools empower users of
the framework to instantly change the underlying kernel without the need to know
the peculiarities of the respective kernels. Using microkernels has never been
easier.
The second challenge of translating each kernel's specific behaviour to the
framework's unified API longs for an automated testing infrastructure that
systematically exercises all the various facets of the API on all base
platforms. The new version introduces the tooling support especially designed
for conducting such quality-assurance measures. These tools largely remove the
burden of manual testing while helping us to uphold the stability and quality
of the framework as it grows in terms of functional complexity and number of
base platforms.
Speaking of functional enhancements, the work on version 11.08 was focused
on our block-device infrastructure and ARM support. The block-device-related
work is primarily motivated by our fundamental goal to scale Genode to a
general-purpose computing platform. The additions comprise new drivers for
SD-cards, IDE, SATA, USB storage as well as a new partition server. All those
components provide Genode's generic block interface, which is meant to be used
as back end for file systems. On file-system level, a new libc plugin utilizes
libffat to enable the straight-forward use of VFAT partitions by libc-using
programs.
The current release comes with far-reaching improvements with respect to
ARM-based platforms. The paravirtualized L4Linux kernel has been updated to
Linux version 2.6.39 running on both x86_32 and ARM. Also, Qt4 including Webkit
has become functional on ARMv6-based platforms.
Among the further improvements are many new examples in the form of
ready-to-use run scripts as well as a comprehensive documentation update.
Originally, we had planned to complement the Noux runtime environment to
support interactive command-line applications by the time of the current
release. However, we realized that the current users of the framework would
value the new streamlined tooling support, the enhanced documentation, and the
new quality-assurance infrastructure over such a functional addition. Hence, we
prioritized the topics accordingly. Even though you will find the first bits of
interactive GNU application support in this release, we deferred working on
this topic in full steam to the upcoming version 11.11.
Blurring the boundaries between different kernels
#################################################
Before the Genode project was born, each microkernel carried along its own
userland. For example, the L4/Fiasco kernel came with the L4 environment, the
OKL4 kernel came with Iguana, or the L4ka::Pistachio kernel came with a small
set of example components. Those user-level counterparts of the kernel
complemented their respective kernels with a runtime for user-level
applications and components while exposing significant parts of the kernel
interface at API level. Consequently, most if not all applications developed
against these APIs were tied to a particular kernel. On the one hand, this
approach enabled developers to fine-tune their programs using kernel-specific
features. On the other hand, much effort was wasted by duplicating other
people's work. Eventually, all of the mentioned userlands stayed limited to
special purposes - for the most part the purposes of operating-systems
researchers. Consequently, none of the microkernels gained much attention in
general-purpose computing. Another consequence of the highly fragmented
microkernel community was the lack of a common ground to compare different
kernels in an unbiased way because each userland provided a different set of
components and libraries.
Different application areas call for different kernel features such as
security mechanisms, scheduling, resource management, and hardware support.
Naturally, each kernel exhibits a specific profile of these parameters
depending on its primary purpose. If one microkernel attempted to accommodate
too many features, it would certainly sacrifice the fundamental idea of being
minimally complex. Consequently, kernels happen to be vastly different. During
the past three years, however, Genode has demonstrated that one carefully
crafted API can target highly diverse kernels, and thereby enables users of
the framework to select the kernel that fits best with the requirements
dictated by each application scenario individually. For us Genode developers,
it was extremely gratifying to see that kernels as different as Linux and NOVA
can be reconciled at the programming-interface level. Still, each kernel comes
with different tools, configuration mechanisms, and boot concepts. Even though
Genode programs can be developed in a kernel-independent way, the deployment of
such programs still required profound insights into the peculiarities of the
respective kernel.
With the current release, we introduce a fundamentally new way of using
different microkernels by unifying the procedures of downloading and building
kernels as well as integrating and running Genode programs with each of them.
Existing Genode application scenarios can be ported between kernels in an
instant without the need for deep insights into the kernel's technicalities. As
a teaser, consider the following commands for building and running Genode's
graphical demo scenario on the OKL4 microkernel:
! # check out Genode
! svn co https://genode.svn.sourceforge.net/svnroot/genode/trunk genode
!
! # download the kernel, e.g., OKL4
! make -C genode/base-okl4 prepare
!
! # create Genode build directory
! genode/tool/create_builddir \
! okl4_x86 BUILD_DIR=build
!
! # build everything and execute the interactive demo
! make -C build run/demo
The same principle steps can be used for any of the OKL4, NOVA,
L4/Fiasco, Fiasco.OC, L4ka::Pistachio, or Codezero kernels. You should
nevertheless consult the documentation at 'base-<platform>/doc/' before
starting to use a specific kernel because some base platforms require
the installation of additional tools.
Under the hood, this seamless way of dealing with different kernels is made
possible by the following considerations:
:Repository preparation:
Each kernel comes from a different source such as a Git/SVN/Mercurial
repository or a packaged archive. Some kernels require additional patches. For
example, OKL4 needs to be patched to overcome problems with modern tool chains.
Now, each 'base-<platform>' repository hosts a 'Makefile' that automates the
download and patch procedure. To download the source code of a kernel,
issue 'make prepare' from within the kernel's 'base-<platform>' directory. The
3rd-party source code will be located at 'base-<platform>/contrib/'.
:Building the kernel:
Each kernel has a different approach when it comes to configuration and
compilation. For example, NOVA comes with a simple 'Makefile', OKL4 relies on a
complex SCons-based build system, L4ka::Pistachio uses CML2 and autoconf (for
the userland tools). Furthermore, some kernels require the setting of specific
configuration values. We have streamlined all these procedures into the Genode
build process by the means of a 'kernel' pseudo target and a 'platform' pseudo
library. The kernel can be compiled directly from the Genode build directory by
issuing 'make kernel'. The 'platform' pseudo library takes care of making the
kernel headers available to Genode. For some kernels such as OKL4 and NOVA, we
replaced the original build mechanism with a Genode target. For other kernels
such as L4ka::Pistachio or Fiasco.OC, we invoke the kernel's build system.
:Genode build directory:
Genode build directories are created via the 'tool/create_builddir' tool.
This tool used to require certain kernel-specific arguments such as the
location of the kernel source tree. Thanks to the unified way of preparing
kernels, the need for such arguments has vanished. Now, the only remaining
arguments to 'create_builddir' are the actual platform and the location
of the build directory to create.
:System integration and booting:
As diverse the build systems of the kernels are, so are the boot concepts. Some
kernels rely on a multiboot-compliant boot loader whereas others have special
tools for creating boot images. Thankfully, Genode's run concept allows us to
hide the peculiarities of booting behind a neat and easy-to-use facade. For
each platform we have crafted a dedicated run environment located at
'base-<platform>/run/env', which contains the rules for system integration and
booting. Therefore, one and the same run script can be used to build and
execute one application scenario across various different kernels. For an
illustrative example, the 'os/src/run/demo.run' script can be executed on all
base platforms (except for base-mb) by issuing 'make run/demo' from within the
build directory.
Emerging block-device infrastructure
####################################
Since version 10.08, Genode is equipped with a block-session interface. Its
primary use cases so far were the supply of the paravirtualized OKLinux kernel
with backing store, and the access of the content of a bootable Live CD.
However, for our mission to use Genode as general-purpose computing platform,
disk device access is crucial. Therefore, we dedicated our attention to
various aspects of Genode's block-device infrastructure, reaching from
programming APIs for block drivers, over partition handling, to file-system
access.
:Block session interface:
The glue that holds all block-device-related components together is the generic
block interface 'os/include/block_session'. It is based on the framework's
packet-stream facility, which allows the communication of bulk data via shared
memory and a data-flow protocol using asynchronous notifications. The interface
supports arbitrary allocation schemes and the use of multiple outstanding
requests. Hence, it is generally suited for scatter-gather DMA and the use of
command queuing as offered by the firmware of modern block-device controllers.
(albeit the current drivers do not exploit this potential yet)
:Block component framework:
Our observation that components implementing the block session interface share
similar code patterns prompted us to design a framework API for implementing
this family of components. The set of classes located at 'os/include/block'
facilitate the separation of device-specific code from application logic.
Whereas 'component.h' provides the application logic needed to implement the
block service, the 'driver.h' is an abstract interface to be implemented by the
actual device driver. This new infrastructure significantly reduces code
duplication among new block-device drivers.
:Device-driver implementations:
The new block-device drivers introduced with the current release address
common types of block devices:
* By adding ATA read/write support to the ATAPI driver ('os/src/drivers/atapi'),
this driver can be used to access IDE disks now.
* The new fully-functional SD-card driver ('os/src/drivers/sdcard') enables the
use of SD-cards connected via the PL180 controller.
* The USB storage driver ('linux_drivers/src/drivers/usb') has been adapted
to the block-session interface and can be used on PC hardware.
* The new AHCI driver ('os/src/drivers/ahci') enables the access of disks
connected via SATA on PC hardware.
Because all drivers are providing the generic block-session interfaces, they
can be arbitrarily combined with components that use this interface as back
end, for example, the partition server and file systems.
:Partition manager as resource multiplexer:
The new partition manager ('os/src/server/part_blk') multiplexes one back-end
block session to multiple block sessions, each accessing a different partition.
Its natural role is being "plugged" between a block-device driver and a file
system.
:File-system access:
Even though a session interface for file systems does not exist yet, we
enabled the use of VFAT partitions through a libc plugin. This libc plugin uses
the ffat library to access files stored on a block device. An
application using this plugin can be directly connected to a block session.
New documentation
#################
The new way of dealing with different kernels motivated us to revisit and
complement our exiting documentation. The following documents are new or
have received considerable attention:
:[https://genode.org/documentation/developer-resources/getting_started - Getting started]:
The revised guide of how to explore Genode provides a quick way to
test drive Genode's graphical demo scenario with a kernel of your
choice and gives pointers to documents needed to proceed your
exploration.
:[https://genode.org/documentation/developer-resources/build_system - Build system manual]:
The new build-system manual explains the concepts behind Genode's
build system, provides guidance with creating custom programs and
libraries, and covers the tool support for the automated integration
and testing of application scenarios.
:[https://genode.org/documentation/components - Components overview]:
The new components-overview document explains the categorization of
Genode's components and lists all components that come with the framework.
:[https://genode.org/documentation/developer-resources/init - Configuration of the init process]:
The document describes Genode's configuration concept, the routing of
service requests, and the expression of mandatory access-control policies.
:[https://genode.org/community/wiki - Wiki]:
The platform-specific Wiki pages for L4/Fiasco, L4ka::Pistachio, NOVA,
Codezero, Fiasco.OC, and OKL4 have been updated to reflect the new flows of
working with the respective base platforms.
Base framework
##############
The RPC API for performing procedure calls across process boundaries
introduced with the version 11.05 was the most significant API change
in Genode's history. To make the transition from the old client-server
API to the new RPC API as smooth as possible, we temporarily upheld
compatibility to the old API. Now, the time has come to put the old
API at rest. The changes that are visible at API level are as follows:
* The old client-server API in the form of 'base/server.h' is no more.
The functionality of the original classes 'Server_entrypoint' and
'Server_activation' is contained in the 'Rpc_entrypoint' class provided
via 'base/rpc_server.h'.
* When introducing the RPC API, we intentionally left the actual session
interfaces as unmodified as possible to proof the versatility of the new
facility. However, it became apparent that some of the original interfaces
could profit from using a less C-ish style. For example, some interfaces used
to pass null-terminated strings as 'char const *' rather than via a dedicated
type. The methodology of using the new RPC API while leaving the original
interfaces intact was to implement such old-style functions as wrappers
around new-style RPC functions. These wrappers were contained in
'rpc_object.h' files, e.g. for 'linux_dataspace', 'parent', 'root',
'signal_session', 'cpu_session'. Now, we have taken the chance to modernise
the API by disposing said wrappers. Thereby, the need for 'rpc_object.h'
files has (almost) vanished.
* The remaining users of the old client-server API have been adapted to the
new RPC API, most prominently, the packet-stream-related interfaces such as
'block_session', 'nic_session', and 'audio_session'.
* We removed 'Typed_capability' and the second argument of the 'Capability'
template. The latter was an artifact that was only used to support the
transition from the old to the new API.
* The 'ipc_client' has no longer an 'operator int'. The result of an IPC can
be requested via the 'result' function.
* We refined the accessors of 'Rpc_in_buffer' in 'base/rpc_args.h'. The
'addr()' has been renamed to 'base()', 'is_valid_string()' considers the
buffer's capacity, and the new 'string()' function is guaranteed to return a
null-terminated string.
* We introduced a new 'Rm_session::Local_addr' class, which serves two
purposes. It allows the transfer of the bit representation of pointers across
RPC calls and effectively removes the need for casting the return type of
'Rm_session::attach' to the type needed at the caller side.
* The 'Connection' class template has been simplified, taking the session
interface as template argument (rather than the capability type). This change
simplified the 'Connection' classes of most session interfaces.
* The never-used return value of 'Parent::announce' has been removed. From the
child's perspective, an announcement always succeeds. The way of how the
announcement is treated is entirely up to the parent. The client should never
act differently depending on the parent's policy anyway.
* The new 'Thread_base::cap()' accessor function allows obtaining the thread's
capability as used for the argument to CPU-session operations.
Operating-system services and libraries
#######################################
Dynamic linker
==============
As a follow-up to the major revision of the dynamic linker that was featured
with the previous release, we addressed several corner cases related to
exception handling and improved the handling of global symbols.
The dynamic linker used to resolve requests for global symbols by handing out
its own symbols if present. However, in some cases, this behaviour is
undesired. For example, the dynamic linker contains a small set of libc
emulation functions specifically for the ported linker code. In the presence of
the real libc, however, these symbols should never be considered at all. To
avoid such ambiguities during symbol resolution, the set of symbols to be
exported is now explicitly declared by the white-list contained in the
'os/src/lib/ldso/symbol.map' file.
We changed the linkage of the C++ support library ('cxx') against dynamic
binaries to be consistent with the other base libraries. Originally, the 'cxx'
library was linked to both the dynamic linker and the dynamic binary, which
resulted in subtle problems caused by the duplication of cxx-internal data
structures. By linking 'cxx' only to the dynamic linker and exporting the
'__cxa' ABI as global symbols, these issues have been resolved. As a positive
side effect, this change reduces the size of dynamic binaries.
C++ exception handling in the presence of shared libraries turned out to be
more challenging than we originally anticipated. For example, the
'_Unwind_Resume' symbol is exported by the compiler's 'libsupc++' as a hidden
global symbol, which can only be resolved when linking this library to the
binary but is not seen by the dynamic linker. This was the actual reason of why
we used to link 'cxx' against both dynamic binaries and shared libraries
causing the problem mentioned in the previous paragraph. Normally, this problem
is addressed by a shared library called 'libgcc_s.so' that comes with the
compiler. However, this library depends on glibc, which prevents us from using
it on Genode. Our solution is renaming the hidden global symbol using a
'_cxx__' prefix and introducing a non-hidden global wrapper function
('__cxx__Unwind_Resume' in 'unwind.cc'), which is resolved at runtime by the
dynamic linker.
Another corner case we identified is throwing exceptions from within the
dynamic linker. In contrast to the original FreeBSD version of the dynamic
linker, which is a plain C program that can never throw a C++ exception,
Genode's version relies on C++ code that makes use of exceptions. To support
C++ exceptions from within the dynamic linker, we have to relocate the
linkers's global symbols again after having loaded the dynamic binary. This
way, type information that is also present within the dynamic binary becomes
relocated to the correct positions.
Block partition server
======================
The new block-partition server uses Genode's block-session interfaces as both
front and back end, leading to the most common use case where this server will
reside between a block driver and a higher level component like a file-system
server.
At startup, the partition server will try to parse the master boot record (MBR)
of its back-end block session. If no partition table is found, the whole block
device is exported as partition '0'. In the other case, the MBR and possible
extended boot records (EBRs) are parsed and offered as separate block sessions
to the front-end clients. The four primary partitions will receive partition
numbers '1' to '4' whereas the first logical partition will be assigned to '5'.
The policy of which partition is exposed to which client can be expressed
in the config supplied to the 'part_blk' server. Please refer to the
documentation at 'os/src/server/part_blk/README' for further details. As an
illustration of the practical use of the 'part_blk' server, you can find a run
script at 'os/run/part_blk.run'.
Skeleton of text terminal
=========================
As part of the ongoing work towards using interactive text-based GNU software
on Genode, we created the first bits of the infrastructure required for
pursuing this quest:
The new terminal-session interface at 'os/include/terminal_session/' is the
designated interface to be implemented by terminal programs.
After investigating the pros and cons of various terminal protocols and
terminal emulators, we settled for implementing a custom terminal emulator
implementing the Linux termcap. This termcap offers a reasonable small set of
commands while providing all essential features such as function-key support
and mouse support. Thanks to Peter Persson for pointing us to the right
direction! The preliminary code for parsing the escape sequences for the Linux
termcap is located at 'gems/include/terminal/'.
We have created a simplistic terminal service that implements the
terminal-session interface using a built-in font. Please note that the
implementation at 'gems/src/server/terminal/' is at an early stage. It is
accompanied by a simple echo program located at 'gems/src/test/terminal_echo'.
Device drivers
##############
USB HID and USB storage
=======================
We replaced the former DDE-Linux-based USB-related driver libraries (at the
'linux_drivers/' repository) by a single USB driver server that offers the
'Input' and 'Block' services. This enables us to use both USB HID and USB
storage at the same time. The new USB driver is located at
'linux_drivers/src/drivers/usb/'.
For using the USB driver as input service (supporting USB HID), add the
'<hid/>' tag to the 'usb_drv' configuration. Analogously, for using the driver
as block service, add the '<storage/>' tag. Both tags can be combined.
For testing the USB stack, the 'linux_drivers' repository comes with the run
scripts 'usb_hid.run' and 'usb_storage.run'.
ATA read/write support
======================
The ATAPI driver has been extended to support IDE block devices for both
read and write transactions. To use the new facility, supply 'ata="yes"'
as XML attribute to the config node of 'atapi_drv'. Please note that this
driver was primarily tested on Qemu. Use it with caution.
SATA driver
===========
The new SATA driver at 'os/src/drivers/ahci/' implements the block-driver
API ('os/include/block'), thus exposing the block-session interface as
front-end. AHCI depends on Genode's PCI driver as well as the timer server. For
a usage example see: 'os/run/ahci.run'.
Limitations and known issues
----------------------------
Currently, the server scans the PCI bus at startup and retrieves the first available
AHCI controller, scans the controller ports and uses the first non-ATAPI port
where a device is present.
On real hardware and on kernels taking advantage of I/O APICs (namely NOVA and
Fiasco.OC) we still lack support for ACPI parsing and thus for interrupts,
leading to a non-working driver.
SD-card driver
==============
The first fragments of our SD-card driver that we introduced with the previous
release have been complemented. The new SD-card driver located at
'os/src/drivers/sd_card/' implements the block-session interface by using
MMC/SD-cards and the PL180 controller as back end. Currently the driver
supports single-capacity SD cards. Therefore, the block file for Qemu should
not exceed 512 MB. Because the driver provides the generic block-session
interface, it can be combined with the new 'libc_ffat' plugin in a
straight-forward way. To give the driver a quick spin, you may give the
'libports/run/libc_ffat.run' script on the 'foc_pbxa9' platform a try.
ARM Realview PL011 UART driver
==============================
The new PL011 UART driver at 'os/src/drivers/uart/' implements the LOG session
interface using the PL011 device. Up to 4 UARTs are supported. The assignment
of UARTs to clients can be defined via a policy supplied to the driver's config
node. For further information, please refer to the README file within the
'uart' directory.
Libraries and applications
##########################
Hello tutorial
==============
The 'hello_tutorial/' repository contains a step-by-step guide for building
a simple client-server scenario. The tutorial has been rewritten for the new
RPC API and is now complemented by a run script for testing the final scenario
on various base platforms.
C and C++ runtimes
==================
:Support for standard C++ headers:
Triggered by public demand for using standard C++ headers for Genode applications,
we introduced a generally usable solution in the form of the 'stdcxx' library
to the 'libc' repository. The new 'stdcxx' library is not a real library. (you
will find the corresponding 'lib/mk/stdcxx.mk' file empty) However, it comes
with a 'lib/import/import-stdcxx.mk' file that adds the compiler's C++ includes
to the default include-search path for any target that has 'stdcxx' listed in
its 'LIBS' declaration.
:Libc back end for accessing VFAT partitions:
The new 'libc_ffat' libc plugin uses a block session via the ffat library. It
can be used by a Genode application to access a VFAT file system via the libc
file API. The file-system access is performed via the 'ffat' library. To
download this library and integrate it with Genode, change to the 'libports'
repository and issue the following command:
! make prepare PKG=ffat
For an example of how to use the libc-ffat plugin, please refer to the run
script 'libports/run/libc_ffat.run'. The source code of the test program can be
found at 'libports/src/test/libc_ffat/'.
Qt4
===
Qt4 version 4.7.1 has been enabled on ARMv6-based platforms, i.e., PBX-A9 on
Fiasco.OC. The support comprises the entire Qt4 framework including qt_webcore
(Webkit).
L4Linux
=======
L4Linux enables the use of one or multiple instances of Linux-based operating
systems as subsystems running on the Fiasco.OC kernel. The Genode version of
L4Linux has seen the following improvements:
:Kernel version: has been updated to Linux 2.6.39.
:ARM support: The L4Linux kernel can be used on ARM-based platforms now.
The PBX-A9 platform is supported via the 'l4linux.run' script as found
at 'ports-foc/run/'. Please find more information at 'ports-foc/README'.
:Genode-specific stub drivers outside the kernel tree:
The stub drivers that enable the use of Genode's services as virtual
devices for L4Linux have been moved outside the kernel patch, which
makes them much easier to maintain. These stub drivers are located
under 'ports-foc/src/drivers/'.
Platform support
################
All base platforms are now handled in a unified fashion. Downloading 3rd-party
source code is performed using the 'prepare' rule of the 'Makefile' provided by
the respective kernel's 'base-<platform>' repository. Once, the platform's base
repository is prepared, the kernel can be built directly from the Genode
build directory using 'make kernel'. All base platforms are now supported by
Genode's run mechanism that automates the tasks of system integration and
testing. For more details about each specific kernel, please revisit the
updated documentation within the respective 'base-<platform>/doc/' directory.
:L4/Fiasco:
The kernel has been updated to revision 472, enabling the use of recent
GNU tool chains.
:Fiasco.OC:
The kernel as been updated to revision 36, which remedies stability problems
related to interaction of the IPC path with thread destruction. The new version
improves the stability of highly dynamic workloads that involve the frequent
creation and destruction of subsystems. However, we experienced the new kernel
version to behave instable on the x86_64 architecture. If you depend on x86_64,
we recommend to temporarily stick with Genode 11.05 and Fiasco.OC revision 31.
:L4ka::Pistachio:
The kernel has been updated to revision 803, enabling the use of recent
versions of binutils.
:OKL4:
OKL4v2 is showing its age. Apparently, the use of the original distribution
requires tools (i.e., python 2.4) that do not ship with current Linux
distributions anymore. This makes it increasingly difficult to use this kernel.
Still, we find ourselves frequently using it for our day-to-day development. To
streamline the use of OKL4v2, we have now incorporated the kernel compilation
into the Genode build system and thereby weakened the kernel's dependency on
ancient tools. However, we decided to drop support for OKL4/ARM for now. We
figured that the supported GTA01 platform is hardly used anymore and hard to
test because it is unsupported by Qemu. Newer ARM platforms are supported by
other kernels anyway.
:Codezero:
Even though B-Labs apparently abandoned the idea of developing the Codezero
kernel in the open, we adapted Genode to the kernel's most recent Open-Source
version that is still available at the official Git repository. Furthermore,
the kernel is now fully supported by Genode's new 'make prepare' procedure and
run environment. Therefore, run scripts such as 'run/demo' can now easily be
executed on Codezero without the need to manually configure the kernel.
Note that, for now, we have disabled Codezero's capabilities because they do
not allow the assignment of device resources. Consequently, 'sys_map' fails for
MMIO regions when performing the capability check (calling 'cap_map_check').
Furthermore, the current version of the kernel requires a workaround for a
current limitation regarding the definition of a thread's pager. At some point,
Codezero abandoned the facility to define the pager for a given thread via the
exregs system call. Instead, the kernel hard-wires the creator of the thread as
the thread's pager. This is conflicting with Genode's way of creating and
paging threads. In the current version of Genode for this kernel, all threads
are paged by one thread (thread 3 happens to be the global pager) within core.
As a workaround to Codezero's current limitation, we define thread 3 to be the
pager of all threads. The patch of the upstream code is automatically being
applied by the 'make prepare' mechanism.
Build system and tools
######################
In addition to the major change with respect to the integration of the various
base platforms, Genode's tool support received the following incremental
improvements:
Build system
============
:Simplification of 'create_builddir' tool:
The 'create_builddir' tool has been relocated from
'tool/builddir/create_builddir' to 'tool/create_builddir' to make it more
readily accessible. Furthermore, we simplified the usage of the tool by
removing the mandatory 'GENODE_DIR' argument. If not explicitly specified, the
tool deduces 'GENODE_DIR' from the its known location within the Genode source
tree.
:Booting from USB sticks:
For most x86-based base platforms, their respective run environments execute
Genode from an ISO image via Qemu. Naturally, such an ISO image can be burned
onto a CD-ROM to be used to boot a real machine. However, booting from CD-ROM
is slow and optical drives are becoming scarce. Therefore we changed the
procedure of creating ISO images to support writing the resulting images to a
USB stick. Under the hood, the boot mechanism chain-loads GRUB via ISOLinux.
The files to implement the boot concept are located at 'tool/boot/'.
:Support for source files in target sub directories:
Until now, the 'SRC_*' declarations in target description files contained
a list of plain file names. The location of the files within the directory
tree had to be defined via 'vpath'. This led to inconveniences when building
3rd-party code that contains files with the same name at different subdirectories.
To resolve such an ambiguity, the target had to be decomposed into multiple
libraries each building a different set of subdirectories. To make the
build system more convenient to use, we have now added support for specifying
source codes with a relative pathname. For example, instead of using
! SRC_CC = main.cc addon.cc
! vpath addon.cc $(PRG_DIR)/contrib
we can now use
! SRC_CC = main.cc contrib/addon.cc
Automated testing across multiple kernels
=========================================
To execute one or multiple test cases on more than one base platform, we
introduced a dedicated tool located at 'tool/autopilot'. Its primary purpose is
the nightly execution of test cases. The tool takes a list of platforms and a
list of run scripts as arguments and executes each run script on each platform.
The build directory for each platform is created at
'/tmp/autopilot.<username>/<platform>' and the output of each run script is
written to a file called '<platform>.<run-script>.log'. On stderr, autopilot
prints the statistics about whether or not each run script executed
successfully on each platform. If at least one run script failed, autopilot
returns a non-zero exit code, which makes it straight forward to include
autopilot into an automated build-and-test environment.

1008
doc/release_notes/11-11.txt Normal file
View File

File diff suppressed because it is too large Load Diff

862
doc/release_notes/12-02.txt Normal file
View File

@@ -0,0 +1,862 @@
===============================================
Release notes for the Genode OS Framework 12.02
===============================================
Genode Labs
The release of Genode 12.02 marks an exciting point in the history of the
project as it is the first version developed in the open rather than within the
chambers of Genode Labs. Thereby, we have embraced GitHub as central facility
for discussion and source-code management. This change has benefits for users
and developers of the framework alike. For users, it has become possible to get
hold of the latest developments using the official 'genodelabs/master' branch and
get involved with discussing the current activities. For regular Genode
developers, the public Git repository replaces a former mix of public
Subversion and company-internal Mercurial repositories, making life much
easier. In Section [Liberation of the development process], we outline the
motivation behind this change and give pointers to the new resources.
The major new additions to the base system are a new framework API for accessing
memory-mapped I/O resources, special support for using Genode as user-level
component framework on Linux, and API support for the reuse of existing
components in the form of sandboxed libraries. These changes are accompanied
with new device-driver infrastructure such as the first version of a device
driver manager and a new ACPI parser.
Feature-wise, the current release takes the first steps towards the goal of the
[https://genode.org/about/road-map - Roadmap for 2012], turning Genode into a
general-purpose OS ready for everyday use by its developers. According to the
roadmap, we enhanced the Noux runtime with fork semantics so that we can run
command-line based GNU programs such as the bash shell and coreutils unmodified
and natively on various microkernels. Furthermore, the library infrastructure
has been enhanced by porting and updating libraries such as Qt 4.7.4 and the
MuPDF PDF rendering engine.
Liberation of the development process
#####################################
In summer 2011, we started a discussion within Genode Labs about changing the
mode of how Genode is developed. Until then, most design discussions and the
actual development work took place locally at the company. At quarterly
intervals, we used to publish our work in the form of official Genode
releases. This way of development seemed to work quite well for us, we were
satisfied about the pace of development, and with each release, our project got
more recognition.
However, the excellent book [https://producingoss.com/ - Producing Open Source Software]
made us realize that even though we released our work under an Open-Source
license, our development process was actually far from being open and may have
discouraged participation of people outside the inner circle of developers.
Because we believe that the framework has reached a state where it becomes
interesting for a wider audience of users and developers, the idea was born
to liberate the project from its closed fashion of development.
In the beginning of December, the vague idea has become a plan. So we finally
brought the topic to our mailing list
([https://genode.org/news/steps-towards-an-open-development-process - Steps towards an open development process]).
We decided to take the release cycle for Genode 12.02 as the opportunity to put
our plan to practice. The central element of this endeavour was moving the
project over to GitHub and adapt our workflows and tooling support accordingly.
First, we started to embrace GitHub's issue tracker for the management of
working topics:
:[https://github.com/genodelabs/genode/issues]: Issue Tracker
The most significant step was leaving our Genode-Labs-internal code
repositories behind and starting a completely public Git repository instead:
:[https://github.com/genodelabs]: Genode Labs at GitHub
With the code repository going public, the way was cleared to opening up design
discussions as well. Instead of having such discussions internally at Genode
Labs, we try to increasingly take them to our mailing list and issue tracker.
With this new way of development, we hope to make the project much more
approachable for people who want to get involved and let Genode reach far out
beyond the reach of our little company.
The changes mentioned above are actually just the tip of the iceberg. For
example, the transition phase required us to rethink the way the project
website is maintained. From now on, almost all of the content of genode.org
comes directly from the project's Git repository. So maintaining website
content is done in the same coherent and transparent way as working on Genode's
code base. So we could finally put the old Wiki to rest. In the process, we
largely revisited the existing content. For example, we rewrote the
[https://genode.org/community/contributions - contributions] document in a
tutorial-like style and incorporated several practical hints, in particular
related to the recommended use of Git.
Although it is probably too early to judge the outcome of our transition, we
are excited how smooth this massive change went. We attribute this pleasant
experience mostly to the excellent GitHub hosting platform, which instantly
ignited a spirit of open collaboration among us. We are excited to see new
people approaching us and showing their interest for teaming up, and we are
curious about where this new model of development will take Genode in the
future.
Base framework, low-level OS infrastructure
###########################################
RPC framework refinements
=========================
Until now, the RPC framework did not support const RPC functions. Rather than
being a limitation inherent to the concept, const RPC functions plainly did not
exist. So supporting them was not deemed too important. However, there are uses
of RPC interfaces that would benefit from a way to declare an RPC function as
const. Candidates are functions like 'Framebuffer::Session::mode()' and
'Input::Session::is_pending()'.
With the current version, we clear the way towards declaring such functions as
const. Even though the change is pretty straight-forward, the thorough support
for const-qualified RPC functions would double the number of overloads for the
'call_member' function template (in 'base/include/util/meta.h'). For this
reason, as of now, the support of const functions is limited to typical getter
functions with no arguments. This appears to be the most common use of such
functions.
API support for enslaving services
==================================
While evolving and using the framework, we always keep an eye on recurring
patterns of how its API is used. Once such a pattern becomes obvious, we try
to take a step back, generalize the observed pattern, and come up with a new
building block that unifies the former repetitive code fragments.
One of those patterns that was far from obvious when we designed Genode years
ago is the use of a service running as child of its own client. At the first
glance, this idea seems counter-intuitive because normally, services are
understood as components that operate independently and protected from their
(untrusted) clients. But there is a class of problems where this approach
becomes extremely useful: The reuse of protocol implementations as a
library-like building block. Most services are actually protocol stacks that
translate a low-level protocol to a more abstract API. For example, a block
device driver translates a specific device API to the generic 'Block_session'
interface. Or the 'iso9660' service translates the 'Block_session' interface to
the 'Rom_session' interface by parsing the ISO9660 file system. Or similarly,
the 'tar_rom' service parses the tar file format to make its content available
via the 'Rom_session' interface.
If a particular functionality is needed by multiple programs, it is common
practice to move this functionality into a dedicated library to avoid the
duplication of the same code at many places. For example, if a program would
need to parse a tar archive, it might be tempting to move the tar-parsing code
from the 'tar_rom' service into a dedicated library, which can then be used by
both the 'tar_rom' service and the new program. An alternative approach is to
just re-use the 'tar_rom' service as a black box and treat it like it was a
library. That is, start the 'tar_rom' service as a child process, supply the
resources the component needs to operate and, in turn, use its API (now in the
form of an RPC interface) to get work done. Because the service is started as a
mere tool at the discretion of its client, we call it *slave*. It turns out
that this idea works exceedingly well in many cases. In a way, it resembles the
Unix philosophy to solve complex problems by combining many small tools each
with a specific purpose. In contrast to the use of libraries, the reuse of
entire components has benefits with regard to fault isolation. Because the
reused functionality is sandboxed within a distinct process, the environment
exposed to this code can be tailored to a rigid subset of the host program's
environment. In the event of a fault within the reused component, the reach of
problem is therefore limited.
On the other hand, we observed that even though this idea works as intended,
implementing the idea for a particular use case involved a good deal of
boiler-plate code where most of this code is needed to define the slave's
environment and resources. Hence, we reviewed the existing occurrences of the
slave pattern and condensed their common concerns into the 'Slave_policy' and
'Slave' classes residing in 'os/include/os/slave.h'. The 'Slave' class is meant
to be used as is. It is merely a convenience wrapper for a child process and
its basic resources. The 'Slave_policy' is meant as a hook for service-specific
customizations. The best showcase is the new 'd3m' component located at
'gems/src/server/d3m'. D3m extensively uses the slave pattern by instantiating
and destroying drivers and file-system instances on-the-fly. A further instance
of this pattern can be found in the new ACPI driver introduced with the current
release.
Support for resizable framebuffers
==================================
The framebuffer-session interface has remained largely untouched since the
original release of Genode in 2008. Back then, we were used to rely on C-style
out parameters in RPC functions. The current RPC framework, however, promotes
the use of a more object-oriented style. So the time has come to revisit the
framebuffer session interface. Instead of using C-style out parameters, the new
'mode()' RPC call returns the mode information as an object of type 'Mode'.
Consequently, mode-specific functions such as 'bytes_per_pixel()' have been
moved to the new 'Framebuffer::Mode' class. The former 'info()' function is
gone.
In addition to the overhaul of the RPC interface, we introduced basic support
for resizable framebuffers. The new 'mode_sigh()' function enables a client to
register a signal handler at the framebuffer session. This signal handler gets
notified in the event of server-side mode changes. Via the new 'release()'
function, the client is able to acknowledge a mode change. By calling it, the
client tells the framebuffer service that it no longer uses the original
framebuffer dataspace. So the server can replace it by a new one. After having
called 'release()', the client can obtain the dataspace for the new mode by
calling 'dataspace()' again.
MMIO access framework
=====================
As the arsenal of native device drivers for Genode grows, we are observing
an increased demand to formalize the style of how drivers are written to
foster code consistency. One particular cause of inconsistency used to be
the way of how memory-mapped I/O registers are accessed. C++ has poor support
for defining bit-accurate register layouts in memory. Therefore, driver code
usually carries along a custom set of convenience functions for reading and
writing registers of different widths as well as a list of bit definitions in
the form of enum values or '#define' statements. To access parts of a register,
the usual pattern is similar to the following example (taken from the pl011
UART driver:
! enum {
! UARTCR = 0x030, /* control register */
! UARTCR_UARTEN = 0x0001, /* enable bit in control register */
! ...
! }
! ...
!
! /* enable UART */
! _write_reg(UARTCR, _read_reg(UARTCR) | UARTCR_UARTEN);
This example showcases two inconveniences: The way the register layout is
expressed and the manual labour needed to access parts of registers. In the
general case, a driver needs to also consider 'MASK' and 'SHIFT' values to
implement access to partial registers properly. This is not just inconvenient
but also error prone. For lazy programmers as ourselves, it's just too easy to
overwrite "undefined" bits in a register instead of explicitly masking the
access to the actually targeted bits. Consequently, the driver may work fine
with the current generation of devices but break with the next generation.
So the idea was born to introduce an easy-to-use formalism for this problem. We
had two goals: First, we wanted to cleanly separate the declaration of register
layouts from the program logic of the driver. The actual driver program should
be free from any intrinsics in the form of bit-masking operations. Second, we
wanted to promote uncluttered driver code that uses names (i.e., in the form of
type names) rather than values to express its operations. The latter goal is
actually achieved by the example above by the use of enum values, but this is
only achieved through discipline. We would prefer to have an API that
facilitates the use of proper names as the most convenient way to express an
operation.
The resulting MMIO API comes in the form of two new header files located at
'base/include/util/register.h' and 'base/include/util/mmio.h'.
Register declarations
~~~~~~~~~~~~~~~~~~~~~
The class templates found in 'util/register.h' provide a means to express
register layouts using C++ types. In a way, these templates make up for
C++'s missing facility to define accurate bitfields. Let's take a look at
a simple example of the 'Register' class template that can be used to define
a register as well as a bitfield within this register:
! struct Vaporizer : Register<16>
! {
! struct Enable : Bitfield<2,1> { };
! struct State : Bitfield<3,3> {
! enum{ SOLID = 1, LIQUID = 2, GASSY = 3 };
! };
!
! static void write (access_t value);
! static access_t read ();
! };
In the example, 'Vaporizer' is a 16-bit register, which is expressed via the
'Register' template argument. The 'Register' class template allows for
accessing register content at a finer granularity than the whole register
width. To give a specific part of the register a name, the 'Register::Bitfield'
class template is used. It describes a bit region within the range of the
compound register. The bit 2 corresponds to true if the device is enabled and
bits 3 to 5 encode the 'State'. To access the actual register, the methods
'read()' and 'write()' must be provided as backend, which performs the access
of the whole register. Once defined, the 'Vaporizer' offers a handy way to
access the individual parts of the register, for example:
! /* read the whole register content */
! Vaporizer::access_t r = Vaporizer::read();
!
! /* clear a bit field */
! Vaporizer::Enable::clear(r);
!
! /* read a bit field value */
! unsigned old_state = Vaporizer::State::get(r);
!
! /* assign new bit field value */
! Vaporizer::State::set(r, Vaporizer::State::LIQUID);
!
! /* write whole register */
! Vaporizer::write(r);
Memory-mapped I/O
~~~~~~~~~~~~~~~~~
The utilities provided by 'util/mmio.h' use the 'Register' template class as
a building block to provide easy-to-use access to memory-mapped I/O registers.
The 'Mmio' class represents a memory-mapped I/O region taking its local base
address as constructor argument. Let's take a simple example to see how it is
supposed to be used:
! class Timer : Mmio
! {
! struct Value : Register<0x0, 32> { };
! struct Control : Register<0x4, 8> {
! struct Enable : Bitfield<0,1> { };
! struct Irq : Bitfield<3,1> { };
! struct Method : Bitfield<1,2>
! {
! enum { ONCE = 1, RELOAD = 2, CYCLE = 3 };
! };
! };
!
! public:
!
! Timer(addr_t base) : Mmio(base) { }
!
! void enable();
! void set_timeout(Value::access_t duration);
! bool irq_raised();
! };
The memory-mapped timer device consists of two registers: The 32-bit 'Value'
register and the 8-bit 'Control' register. They are located at the MMIO offsets
0x0 and 0x4, respectively. Some parts of the 'Control' register have specific
meanings as expressed by the 'Bitfield' definitions within the 'Control'
struct.
Using these declarations, accessing the individual bits becomes almost a
verbatim description of how the device is used. For example:
! void enable()
! {
! /* access an individual bitfield */
! write<Control::Enable>(true);
! }
!
! void set_timeout(Value::access_t duration)
! {
! /* write complete content of a register */
! write<Value>(duration);
!
! /* write all bitfields as one transaction */
! write<Control>(Control::Enable::bits(1) |
! Control::Method::bits(Control::Method::ONCE) |
! Control::Irq::bits(0));
! }
!
! bool irq_raised()
! {
! return read<Control::Irq>();
! }
In addition to those basic facilities, further noteworthy features of the new
API are the support for register arrays and the graceful overflow handling
with respect to register and bitfield boundaries.
C Runtime
=========
We extended our FreeBSD-based C runtime to accommodate the needs of the Noux
runtime environment and our port of the MuPDF application.
* The dummy implementation of '_ioctl()' has been removed. This function is
called internally within the libc, i.e., by 'tcgetattr()'. For running
libreadline in Noux, we need to hook into those ioctl operations via the
libc plugin interface.
* The 'libc/regex' and 'libc/compat' modules have been added to the libc.
These libraries are needed by the forthcoming port of Slashem to Noux.
* We added a default implementation of 'chdir()'. It relies on the sequence of
'open()', 'fchdir()', 'close()'.
* The new libc plugin 'libc_rom' enables the use of libc file I/O functions
to access ROM files as provided by Genode ROM session.
* We changed the libc dummy implementations to always return ENOSYS. Prior
this change, 'errno' used to remain untouched by those functions causing
indeterministic behaviour of code that calls those functions, observes the
error return value (as returned by most dummies), and evaluates the error
condition reported by errno.
* If using the libc for Noux programs, the default implementations of
time-related functions such as 'gettimeofday()' cannot be used because they
rely on a dedicated timeout-scheduler thread. Noux programs, however, are
expected to contain only the main thread. By turning the functions into weak
symbols, we enabled the noux libc-plugin to provide custom implementations.
DDE Kit
=======
Linux DDE used to implement Linux spin locks based on 'dde_kit_lock'. This
works fine if a spin lock is initialized only once and used from then on. But
if spin locks are initialized on-the-fly at a high rate, each initialization
causes the allocation of a new 'dde_kit_lock'. Because in contrast to normal
locks, spinlocks cannot be explicitly destroyed, the spin-lock emulating locks
are never freed. To solve the leakage of locks, we complemented DDE Kit with
the new 'os/include/dde_kit/spin_lock.h' API providing the semantics as
expected by Linux drivers.
Libraries and applications
##########################
New and updated libraries
=========================
:Qt4 updated to version 4.7.4:
We updated Qt4 from version 4.7.1 to version 4.7.4. For the most part, the
update contains bug fixes as detailed in the release notes for the versions
[https://qt.nokia.com/products/changes/changes-4.7.2 - 4.7.2],
[https://qt.nokia.com/products/changes/changes-4.7.3 - 4.7.3], and
[https://labs.qt.nokia.com/2011/09/01/qt-4-7-4-released - 4.7.4].
:Update of zlib to version 1.2.6:
Because zlib 1.2.5 is no more available at zlib.net, we bumped the zlib
version to 1.2.6.
:New ports of openjpeg, jbig2dec, and mupdf:
MuPDF is a fast and versatile PDF rendering library with only a few
dependencies. It depends on openjpeg (JPEG2000 codec) and jbig2dec (b/w image
compression library). With the current version, we integrated those libraries
alongside the MuPDF library to the 'libports' repository.
GDB monitor refinements and automated test
==========================================
We improved the support for GDB-based user-level debugging as introduced with
the previous release.
For the x86 architecture, we fixed a corner-case problem with using the
two-byte 'INT 0' instruction for breakpoints. The fix changes the breakpoint
instruction to the single-byte 'HLT'. 'HLT' is a privileged instruction and
triggers an exception when executed in user mode.
The new 'gdb_monitor_interactive.run' script extends the original
'gdb_monitor.run' script with a startup sequence that automates the
initial break-in at the 'main()' function of a dynamically linked binary.
The revised 'gdb_monitor.run' script has been improved to become a full
automated test case for GDB functionalities. It exercises the following
features (currently on Fiasco.OC only):
* Breakpoint in 'main()'
* Breakpoint in a shared-library function
* Stack trace when not in a syscall
* Thread info
* Single stepping
* Handling of segmentation-fault exception
* Stack trace when in a syscall
PDF viewer
==========
According to our road map for 2012, we pursued the port of an existing PDF
viewer as native application to Genode.
We first looked at the [https://poppler.freedesktop.org - libpoppler],
which seems to be the most popular PDF rendering engine in the world of
freedesktop.org. To get a grasp on what the porting effort of this engine may
be, we looked at projects using this library as well as the library source
code. By examining applications such as the light-weight epdfview
application, we observed that libpoppler's primary design goal is to integrate
well with existing freedesktop.org infrastructure rather than to reimplement
functionality that is provided by another library. For example, fontconfig is
used to obtain font information and Cairo is used as rendering backend. In the
context of freedesktop.org, this makes perfect sense. But in our context,
porting libpoppler would require us to port all this infrastructure to Genode
as well. To illustrate the order of magnitude of the effort needed, epdfview
depends on 65 shared libraries. Of course, at some point in the future, we will
be faced to carry out this porting work. But for the immediate goal to have a
PDF rendering engine available on Genode, it seems overly involved. Another
criterion to evaluate the feasibility of integrating libpoppler with Genode is
its API. By glancing at the API, it seems to be extremely feature rich and
complex - certainly not a thing to conquer in one evening with a glass of wine.
The Qt4 backend of the library comprises circa 8000 lines of code. This value
can be taken as a vague hint at the amount of work needed to create a custom
backend, i.e., for Genode's framebuffer-session interface.
Fortunately for us, there exists an alternative PDF rendering engine named
MuPDF. The concept of MuPDF is quite the opposite of that of libpoppler.
MuPDF tries to be as self-sufficient as possible in order to be suitable
for embedded systems without comprehensive OS infrastructure. It comes with a
custom vector-graphics library (instead of using an existing library such as
Cairo) and it even has statically linked-in all font information needed to
display PDF files that come with no embedded fonts. That said, it does not
try to reinvent the wheel in every regard. For example, it relies on
common libraries such as zlib, libpng, jpeg, freetype, and openjpeg. Most
of them are already available on Genode. And the remaining libraries are rather
free-standing and easy to port. To illustrate the low degree of dependencies,
the MuPDF application on GNU/Linux depends on only 15 shared libraries. The
best thing about MuPDF from our perspective however, is its lean and clean API,
and the wonderfully simple example application. Thanks to this example, it was
a breeze to integrate the MuPDF engine with Genode's native framebuffer-session
and input-session interfaces. The effort needed for this integration work lies
in the order of less than 300 lines of code.
At the current stage, the MuPDF rendering engine successfully runs on Genode
in the form of a simple interactive test program, which can be started
via the 'libports/run/mupdf' run script. The program supports the basic key
handling required to browse through a multi-page PDF document
(page-up or enter -> next page, page-down or backspace -> previous page).
Improved terminal performance
=============================
The terminal component used to make all intermediate states visible to the
framebuffer in a fully synchronous fashion. This is an unfortunate behaviour
when scrolling through large text outputs. By decoupling the conversion of the
terminal state to pixels from the 'Terminal::write()' RPC function,
intermediate terminal states produced by sub sequential write operations do not
end up on screen one by one but only the final state becomes visible. This
improvement drastically improves the speed in situations with a lot of
intermediate states.
Noux support for fork semantics
===============================
Genode proclaims to be a framework out of which operating systems can be built.
There is no better way of putting this claim to the test than to use the
framework for building a Unix-like OS. This is the role of the Noux runtime
environment.
During the past releases, Noux evolved into a runtime environment that is able
to execute complex command-line-based GNU software such as VIM with no
modification. However, we cannot talk of Unix without talking about its
fundamental concept embodied in the form of the 'fork()' system call. We did
not entirely dismiss the idea of implementing 'fork()' into Noux but up to now,
it was something that we willingly overlooked. However, the primary goal of
Noux is to run the GNU userland natively on Genode. This includes a good deal
of programs that rely on fork semantics. We could either try to change all the
programs to use a Genode-specific way of starting programs or bite in the
bullet and implement fork. With the current release, we did the latter.
The biggest challenge of implementing fork was to find a solution that is not
tied to one kernel but one that works across all the different base platforms.
The principle problem of starting a new process in a platform-independent
manner is already solved by Genode in the form of the 'Process' API. But this
startup procedure is entirely different from the semantics of fork. The key to
the solution was Genode's natural ability to virtualize the access to low-level
platform resources. To implement fork semantics, all Noux has to do is to
provide local implementations of core's RAM, RM, and CPU session interfaces.
The custom implementation of the CPU session interface is used to tweak the
startup procedure as performed by the 'Process' class. Normally, processes
start execution immediately at creation time at the ELF entry point. For
implementing fork semantics, however, this default behavior does not work.
Instead, we need to defer the start of the main thread until we have finished
copying the address space of the forking process. Furthermore, we need to start
the main thread at a custom trampoline function rather than at the ELF entry
point. Those customizations are possible by wrapping core's CPU service.
The custom implementation of the RAM session interface provides a pool of RAM
shared by Noux and all Noux processes. The use of a shared pool alleviates the
need to assign RAM quota to individual Noux processes. Furthermore, the custom
implementation is needed to get hold of the RAM dataspaces allocated by each
Noux process. When forking a process, the acquired information is used to
create a shadow copy of the forking address space.
Finally, a custom RM service implementation is used for recording all RM
regions attached to the region-manager session of a Noux process. Using the
recorded information, the address-space layout can then be replayed onto a new
process created via fork.
With the virtualized platform resources in place, the only puzzle piece that
is missing is the bootstrapping of the new process. When its main thread is
started, it has an identical address-space content as the forking process but
it has to talk to a different parent entrypoint and a different Noux session.
The procedure of re-establishing the relationship of the new process to its
parent is achieved via a small trampoline function that re-initializes the
process environment and then branches to the original forking point via
setjmp/longjmp. This mechanism is implemented in the libc_noux plugin.
As the immediate result of this work, a simple fork test can be executed across
all base platforms except for Linux (Linux is not supported yet). The test
program is located at 'ports/src/test/noux_fork' and can be started with the
'ports/run/noux_fork.run' script.
Furthermore, as a slightly more exciting example, there is a run script for
running a bash shell on a tar file system that contains coreutils. By starting
the 'ports/run/noux_bash.run' script, you get presented an interactive bash
shell. The shell is displayed via the terminal service and accepts user input.
It allows you to start one of the coreutils programs such as ls or cat. Please
note that the current state is still largely untested, incomplete, and flaky.
But considering that Noux is comprised of less than 2500 lines of code, we are
quite surprised of how far one can get with such little effort.
Device drivers
##############
Driver improvements to accommodate dynamic (re-)loading
=======================================================
To support the dynamic probing of devices as performed by the new d3m
component, the ATAPI and USB device drivers have been enhanced to support the
subsequent closing and re-opening of sessions.
First bits of the d3m device-driver manager
===========================================
The abbreviation d3m stands for demo device-driver manager. It is our current
solution for the automated loading of suitable drivers as needed for running
Genode from a Live CD or USB stick. Because of the current narrow focus of d3m,
it is not a generic driver-management solution but a first step in this
direction. We hope that in the long run, d3m will evolve to become a generic
driver-management facility so that we can remove one of the "D"s from its name.
In the current form d3m solves the problems of merging input-event streams,
selecting the boot device, and dealing with failing network drivers.
When using the live CD, we expect user input to come from USB HID devices or
from a PS/2 mouse and keyboard. The live system should be operational if at
least one of those sources of input is available. In the presence of multiple
sources, we want to accumulate the events of all of them.
The live CD should come in the form of a single ISO image that can be burned onto a CDROM or
alternatively copied to an USB stick. The live system should boot fine in both
cases. The first boot stage is accommodated by syslinux and the GRUB boot
loader using BIOS functions. But once Genode takes over control, it needs to
figure out on its own from where to fetch data. A priori, there is no way to
guess whether the ATAPI driver or the USB storage driver should be used.
[image d3m_what_next]
Therefore, d3m implements a probing mechanism that starts each of the drivers,
probes for the presence of a particular file on an iso9660 file system.
[image d3m_probing]
Once d3m observes a drivers that is able to successfully access the magic file,
it keeps the driver and announces the driver's service to its own parent. For
the system outside of d3m, the probing procedure is completely transparent. D3m
appears to be just a service that always provides the valid block session for
the boot medium.
[image d3m_ready]
The network device drivers that we ported from the iPXE project cover the
range of most common wired network adaptors, in particular the E1000 family.
But we cannot presume that a computer running the live system comes equipped
with one of the supported devices. If no supported network card could be
detected the driver would simply fail. Applications requesting a NIC session
would block until a NIC service becomes available, which won't happen. To
prevent this situation, d3m wraps the NIC driver and provides a dummy NIC
service in the event the drivers fails. This way, the client application won't
block infinitely but receive an error on the first attempt to use the NIC.
ACPI support
============
To accommodate kernels like Fiasco.OC or NOVA that take advantage of x86's
APIC, we have introduced a simple ACPI parser located at 'os/src/drivers/acpi'.
The server traverses the ACPI tables and sets the interrupt line of devices
within the PCI config space to the GSIs found in the ACPI tables. Internally it
uses Genode's existing PCI driver as a child process for performing PCI access
and, in turn, announces a PCI service itself.
For obtaining the IRQ routing information from the ACPI tables without
employing a full-blown ACPI interpreter, the ACPI driver uses an ingenious
technique invented by Bernhard Kauer, which is described in the following
paper:
:[https://os.inf.tu-dresden.de/papers_ps/tr-atare-2009.pdf - ATARE - ACPI Tables and Regular Expressions]:
_TU Dresden technical report TUD-FI09-09, Dresden, Germany, August 2009_
:Usage:
Start the 'acpi_drv' in your Genode environment. Do not start the 'pci_drv'
since this will be used as a slave of the 'acpi_drv'. You still must load the
'pci_drv' in your boot loader. To integrate the ACPI driver into your boot
configuration, you may take the following snippet as reference:
!<start name="acpi">
! <resource name="RAM" quantum="2M"/>
! <binary name="acpi_drv"/>
! <provides><service name="PCI"/></provides>
! <route>
! <service name="ROM"> <parent/> </service>
! <any-service> <any-child/> <parent/> </any-service>
! </route>
!</start>
:Limitations and known issues:
Currently there is no interface to set the interrupt mode for core's IRQ
sessions (e.g., level or edge triggered). This is required by Fiasco.OCs kernel
interface. We regard this as future work.
Platform support
################
Fiasco.OC microkernel
=====================
The support for the Fiasco.OC base platform is still lacking proper handling
for releasing resources such as kernel capabilities. Although this is a known
issue, we underestimated the reach of the problem when Genode's signal API is
used. Each emitted signal happens to consume one kernel capability within core,
ultimately leading to a resource outage when executing signal-intensive code
such as the packet-stream interface. The current release comes with an interim
solution. To remedy the acute problem, we extended the 'Capability_allocator'
class with the ability to register the global ID of a Genode capability so
that the ID gets associated with a process-local kernel capability. Whenever
a Genode capability gets unmarshalled from an IPC message, the
capability-allocator is asked, with the global ID as key, whether the
kernel-cap already exists. This significantly reduces the waste of
kernel-capability slots.
To circumvent problems of having one and the same ID for different kernel
objects, the following problems had to be solved:
* Replace pseudo IDs with unique ones from core's badge allocator
* When freeing a session object, free the global ID _after_ unmapping
the kernel object, otherwise the global ID might get re-used in some
process and the registry will find a valid but wrong capability
for the ID
Because core aggregates all capabilities of all different processes, its
capability registry needs much more memory compared to a regular process.
By parametrizing capability allocators differently for core and non-core
processes, the global memory overhead for capability registries is kept
at a reasonable level.
Please note that this solution is meant as an interim fix until we have
resolved the root of the problem, which is the proper tracking and releasing
of capability selectors.
Linux
=====
Linux is one of the two original base platforms of Genode. The original
intension behind supporting Linux besides a microkernel was to facilitate
portability of the API design and to have a convenient testing environment for
platform-independent code. Running Genode in the form of a bunch of plain Linux
processes has become an invaluable feature for our fast-paced development.
To our delight, we lately discovered that the use of running Genode on Linux
can actually go far beyond this original incentive. Apparently, on Linux, the
framework represents an equally powerful component framework as on the other
platforms. Hence, Genode has the potential to become an attractive option for
creating complex component-based user-level software on Linux.
For this intended use, however, the framework has to fulfill the following
additional requirements:
* Developers on Linux expect that their components integrate seamlessly with
their existing library infrastructure including all shared libraries
installed on Linux.
* The use of a custom tool chain is hard to justify to developers who regard
Genode merely as an application framework. Hence, a way to use the normal
tool chain as installed on the Linux host system is desired.
* Application developers are accustomed with using GDB for debugging and expect
that GDB can be attached to an arbitrary Genode program in an intuitive way.
Genode's original support for Linux as base platform did not meet those
expectations. Because Genode's libc would ultimately collide with the Linux
glibc, Genode is built with no glibc dependency at all. It talks to the kernel
directly using custom kernel bindings. In particular, Genode threads are created
directly via the 'clone()' system call and thread-local storage (TLS) is managed
in the same way as for the other base platforms. This has two implications.
First, because Genode's TLS mechanism is different than the Linux TLS
mechanism, Genode cannot be built with the normal host tool chain. This
compiler would generate code that would simply break on the first attempt to
use TLS. We solved this problem with our custom tool chain, which is configured
for Genode's needs. The second implication is that GDB is not able to handle
threads created differently than those created via the pthread library. Even
though GDB can be attached to each thread individually, the debugger would not
correctly handle a multi-threaded Genode process as a multi-threaded Linux
program. With regard to the use of Linux shared libraries, Genode used to
support a few special programs that used both the Genode API and Linux
libraries. Those programs (called hybrid Linux/Genode programs) were typically
pseudo device drivers that translate a Linux API to a Genode service. For
example, there exists a framebuffer service that uses libSDL as back end.
Because those programs were a rarity, the support by the build system for such
hybrid targets was rather poor.
Fortunately, all the problems outlined above could be remedied pretty easily.
It turns out that our custom libc is simply not relevant when Genode is
used as plain application framework on Linux. For this intended use, we always
want to use the host's normal libc. This way, the sole reason for using plain
system calls instead of the pthread library vanishes, which, in turn,
alleviates the need for a custom tool chain. Genode threads are then simply
pthreads compatible with the TLS code as emitted by the host compiler and
perfectly recognised by GDB. With the surprisingly little effort of creating a
new implementation of Genode's thread API to target pthreads instead of using
syscalls, we managed to provide a decent level of support for using Genode as
user-level component framework on Linux.
These technical changes alone, however, are not sufficient to make Genode
practical for real-world use. As stated above, the few hybrid Linux/Genode
programs used to be regarded as some leprous artifacts. When using Genode as
Linux application framework, however, this kind of programs are becoming the
norm rather than an exception. For this reason, we introduced new support for
such hybrid programs into the build system. By listing the 'lx_hybrid'
library in the 'LIBS' declaration of a target, this target magically becomes a
hybrid Linux/Genode program. It gets linked to the glibc and uses pthreads
instead of direct syscalls. Furthermore, host libraries can be linked to the
program by stating their respective names in the 'LX_LIBS' variable. For an
example, please refer to the libSDL-based framebuffer at
'os/src/drivers/framebuffer/sdl/target.mk'.
To enforce the build of all targets as hybrid Linux/Genode programs, the build
system features the 'alyways_hybrid' 'SPEC' value. To make it easy to create a
build directory with all targets forced to be hybrid, we have added the new
'lx_hybrid_x86' platform to the 'create_builddir' tool.
OKL4
====
:Sending an invalid-opcode exception IPC on OKL4:
When an invalid opcode gets executed, OKL4 switches to the kernel debugger
console instead of sending an exception IPC to the userland. We fixed this
problem by removing the code that invokes the debugger console from the kernel.
:Hard-wire OKL4 elfweaver to Python 2:
Recent Linux distributions use Python version 3 by default. But OKL4's
elfweaver is not compatible with this version of Python. For this reason, we
introduced a patch for pinning the Python version used by elfweaver to
version 2.
Both patches get automatically applied when preparing the 'base-okl4'
repository via 'make prepare'.
Build system and tools
######################
:Facility for explicitly building all libraries:
During its normal operation, the build system creates libraries as mere side
effects of building targets. There is no way to explicitly trigger the build of
libraries only. However, in some circumstances (for example for testing the
thorough build of all libraries), a mechanism for explicitly building libraries
would be convenient. Hence we introduced this feature in the form of the pseudo
target located at 'base/src/lib/target.mk'. By issuing 'make lib' in a build
directory, this target triggers the build of all libraries available for the
given platform.

1029
doc/release_notes/12-05.txt Normal file
View File

File diff suppressed because it is too large Load Diff

665
doc/release_notes/12-08.txt Normal file
View File

@@ -0,0 +1,665 @@
===============================================
Release notes for the Genode OS Framework 12.08
===============================================
Genode Labs
With Genode 12.08, the project focused on platform support. It enters the world
of OMAP4-based ARM platforms, revived and vastly enhanced the support for the
NOVA hypervisor, and becomes able to run directly on ARM platforms without the
need for an underlying kernel.
The new 'base-hw' platform is a deviation from Genode's traditional approach to
complement existing kernels with user-land infrastructure. It completely leaves
the separate kernel out of the picture and thereby dwarfs the base line of the
trusted computing base of Genode-based systems to approximately the half. The
new base platform is described in Section [Genode on naked ARM hardware].
Speaking of base platforms, we are happy to have promoted the NOVA hypervisor
to a first-class citizen among the base platforms. During the last months, this
kernel underwent fundamental changes regarding its mode of development and its
feature set. This prompted us to vastly improve Genode's support for this
platform and leverage its unique features. If considering the use of Genode on
x86-based hardware, NOVA has become a very attractive foundation. Section
[Embracing the NOVA Hypervisor] describes the NOVA-specific changes.
The improvement of platform support with the current release does not entail
the base platforms only but extends to profound additions of device drivers, in
particular for the ARM-based OMAP4 SoC as used on the popular Pandaboard. We
are proud to announce the availability of device drivers for HDMI output,
SD-card, USB HID, and networking for this platform.
Beyond the low-level platform improvements, the new version comes with several
new services, optimizations of existing components, and new ported libraries.
In particular, the Noux runtime has reached a point where we can principally
execute serious networking applications such as the Lynx web browser natively
on Genode. Another example is the new FFAT-based file-system service, which
makes persistent storage available via Genode's file-system interface. By
combining this new service with existing components such as the partition
service, Noux, or the file-system plugin of the libc, a lot of new application
scenarios become available. Thanks to these new components, the framework has
become able to perform on-target debugging via GDB running in Noux, or host
the genode.org website via the lighttpd web server,
:What about the road map?:
Those of you who track the milestones laid out in our [https://genode.org/about/road-map - road map]
may wonder how Genode 12.08 relates to the stated goals. In fact, several
points of the road map haven't received the attention as originally planned.
As an explanation, let us quote the paragraph right atop of the road-map page:
"The road map is not fixed. If there is commercial interest of pushing the
Genode technology to a certain direction, we are willing to revisit our plans."
Well, this is what happened. So we traded the work on the tiled window manager,
the Intel wireless driver, and SMP support for the work on the platform topics
outlined above. Nevertheless, we stick to our overall plan to turn Genode into
a general-purpose OS that is fit for use by its developers by the end of the
year. If looking closely at the additions that come with the current release,
it will become apparent how well they fit into the big picture.
Genode on naked ARM hardware
############################
One of Genode's most distinguishing properties is the ability to use the framework
on top of a range of different kernels. This way, users of the framework
benefit from the wide variety of features provided by those kernels while
only dealing with a single API and configuration concept. For example, we
frequently find ourselves using the Linux kernel as base platform while
developing services, interfaces, and protocol stacks. By being able to start
Genode as a regular program, we effectively eliminate the reboot-time for each
test run and enjoy using commodity debugging and profiling tools. On the other
hand, if high security is a concern, NOVA and Fiasco.OC provide
capability-based security at kernel-level. So the use of one of those kernels
is desirable. Genode allows for switching between those vastly different
kernels almost seamlessly.
In general, a Genode system consists of a kernel, Genode's core, and the
largely generic components on top of core. Core abstracts away the
peculiarities of the respective kernel and provides a unified API to the
components on top. From the application's point of view both kernel and core
are always at the root of the process tree and thereby are a inherent part of
the application's trusted computing base (TCB). The distinction of both
programs is almost superficial.
Since both the kernel and core must be ultimately trusted, the complexity of
both programs is critical for each Genode-based system. On our quest for
minimizing the TCB complexity so far, however, we did not question the role of
the kernel as an inherent part of the TCB and focused our attention to the
software stack on top. However, with more and more kernels entering the
picture, we identified that there is typically a considerable overlap in
functionality between kernel and core. For example, both need to know about
address spaces and their relationship to physical memory objects. Most kernels
keep track of memory mappings in an in-kernel database. Core also needs to keep
track of this information. Consequently, we found several information
replicated without a clear benefit. With this comes a seemingly significant
redundancy of code for data structures, allocators, and utility functions.
Furthermore, there exists a class of problems that must be solved by the kernel
and core alike. In particular the resource management of dynamically allocated
in-kernel objects respectively in-core objects. Whereas core uses Genode's
resource-trading concept to solve this problem, most kernels lack a good
solution for the management of in-kernel resources and are consequently prone
to resource exhaustion problems.
Out of these observations, the idea was born to explore the opportunities of
merging both programs into one and thereby eliminating the redundancies. Our
first attempt to go into this direction was the 'base-mb' platform, which
enabled us to run Genode on the Xilinx MicroBlaze softcore CPU. With this
experiment, we gained confidence that the approach is generally feasible. So we
took on the challenge to implement the idea of a hybrid kernel/core on a more
complex architecture namely ARM Cortex-A9.
The 'base-hw' platform introduced with the current release is the intermediate
result of our experiment. With this base platform, core plays the role of core
and the kernel within one program. A few code paths that require execution in
privileged mode are executed in kernel mode whereas most code paths are
executed in user mode. Both user mode code and kernel mode code run in the same
address space. The kernel portion merely provides a few basic mechanisms
without performing complex operations such as dynamic memory allocations. For
example, if core is requested to create a new thread via core's CPU session
interface, the user-level code within core allocates a KTCB (kernel thread
control block) and UTCB (user-level thread-control block) from the physical
memory allocator and passes both physical addresses to the kernel function that
spawns the actual thread. This way, we can employ Genode's resource-trading
concept for managing typical kernel resources.
The experiment turned out to be a great success. Traditionally, we would account
at least 10,000 lines of code (LOC) for the kernel. Most kernels are actually
much larger than that. Core comes at a complexity of another 10,000 LOC. So
both kernel and core make up a base line of TCB complexity of more than 20,000
LOC. By co-locating core with the kernel, we end up with a program of just
about 13,000 LOC. The vast reduction of TCB complexity compared to having
kernel and core as separate programs strikes us.
The 'base-hw' version of core supports the complete Genode API covering support
for user-level device drivers, synchronous RPCs, asynchronous notifications,
shared memory, and managed dataspaces. It is thereby able to execute the
sophisticated Genode scenarios on top including the GUI, the dynamic linker,
and user-level device drivers. That said, we regard the current version still
as work in progress. We successfully use it as an experimentation platform for
ongoing research activities (i.e., for exploring ARM TrustZone) but some
important features such as capability-based security are not yet implemented.
:Using the base-hw platform:
The new base platform is fully integrated with Genode's build system.
When listing the supported base platforms via the 'tool/create_builddir' tool,
you will see the new 'hw_panda_a2', 'hw_vea9x4', 'hw_pbxa9' choices of
build-directory templates. The latter platform enables you to run a
'base-hw' Genode system on Qemu.
[https://genode.org/documentation/platforms/hw - Learn more about using the new base-hw platform...]
For running Genode directly on the Pandaboard, please refer to the
[https://genode.org/documentation/platforms/hw_panda_a2 - Pandaboard-specific documentation...]
Embracing the NOVA Hypervisor
#############################
NOVA is a so-called microhypervisor for the x86 architecture. It combines the
principles of microkernels with capability-based security and hardware-assisted
virtualization. Among the various base platforms supported by Genode, NOVA's
kernel interface stands out for being extremely minimalistic and orthogonal,
even by microkernel standards.
Genode has supported NOVA as base platform since 2010. But because we used NOVA
solely for sporadic research activities and NOVA's lack of a regular release
schedule, the framework's platform support received only little attention. This
has changed now. NOVA's main developer Udo Steinberg moved from TU Dresden to
Intel Labs where he leads the development of NOVA as a true Open-Source
project. In fact, the code is now being hosted at GitHub:
:[https://github.com/IntelLabs/NOVA]:
NOVA hypervisor at GitHub
Since its move to GitHub, the hypervisor has already seen significant
improvements. The repository is continuously updated, which enables us to stay
in a close feedback loop with the NOVA developers. This change of how NOVA's
development is conducted ignited our renewed interest in promoting this
platform to a first-level citizen of our framework. The first noteworthy
improvement is the recently added 64-bit support of NOVA. We enabled Genode to
work with both variants of the kernel - 32 bit and 64 bit.
But this was just the first step. The second major change addresses the
allocation of kernel resources. Early versions of the hypervisor allowed each
process to create kernel objects and thereby indirectly consume the limited
memory resources of the kernel. This is perfectly fine for a research project
but it becomes a potential denial-of-service problem in real-world use cases.
For this reason, newer versions introduced the ability to retain the allocation
of kernel objects within a trusted component only. In the Genode world, this
component is naturally core. Even though NOVA still lacks a flexible concept for
kernel-resource management as of now, Genode has become able to use NOVA
without suffering the inherent resource management limitation. This is achieved
because core is able to arbitrate the allocation of kernel resources.
The third fundamental change is the abolishment of the last traces of global
names in a NOVA-based Genode system. There are no thread IDs, object IDs, or
any other kind of globally meaningful names. Each process has a local view on
(a small part of) the system only. If a process interacts with another process,
the kernel translates the references to remote objects from one namespace to
the other. The security implications are eminent. First, a process can only
interact with or refer to objects for which it has a name, which vastly reduces
problems of ambient authority. Second, because the kernel translates names, it
becomes impossible to forge object identities. If a process tried to pass a
forged object reference to another process, the translation would simply fail,
rendering the attack ineffective.
The described changes do not come without issues, though. To make the NOVA
kernel fit with Genode's requirements, minor patches of the hypervisor are
needed. The patches are located at 'base-nova/patches/'. However, those patches
are meant as interim solutions until we find mechanisms that fit well with the
design of the hypervisor and also fulfil our requirements.
So far, we greatly enjoyed the revived collaboration with the NOVA developers
and congratulate Udo Steinberg for the new mode of development of the
hypervisor.
Base framework
##############
In the following, we describe changes of the base API that may affect users of
the framework.
:Allocation of DMA buffers:
We extended the RAM session interface with the ability to allocate DMA buffers.
The client specifies the type of RAM dataspace to allocate via the new 'cached'
argument of the 'Ram_session::alloc()' function. By default, 'cached' is true,
which corresponds to the common case and the original behavior. When setting
'cached' to 'false', core takes the precautions needed to register the memory
as uncached in the page table of each process that has the dataspace attached.
Currently, the support for allocating DMA buffers is implemented for Fiasco.OC
only. On x86 platforms, it is generally not needed. But on platforms with more
relaxed cache coherence (such as ARM), user-level device drivers should always
use uncacheable memory for DMA transactions.
:MMIO framework improvements:
As we find ourselves increasingly using the 'Register' and 'Mmio' templates
provided by 'util/register.h' and 'util/mmio.h' for dealing with memory-mapped
devices, we extended the utilities with support for 64-bit registers and a new
interface for polling bit states. The latter functionality is provided by the
new 'wait_for' function template. To decouple the MMIO-related utility code
from an actual timer facility, the function takes a so-called 'delayer' functor
as argument. This way the user of the MMIO framework is able to pick a timer
facility that fits best with the device.
:New 'memcpy' implementation:
The memory-copy functions provided by 'util/string.h' are extremely simple
and arguably slow, particularly on platforms where byte-wise copy operations
are not supported by the CPU (i.e., ARM). Hence, we have added a CPU-specific
memcpy function ('memcpy_cpu') to 'cpu/string.h', which enables us to
provide optimized implementations. So far, we did so for the ARM architecture.
Low-level OS infrastructure
###########################
FFat-based file-system service
==============================
With the previous release, we introduced Genode's file-system interface
accompanied with a simple in-memory file-system service. With the addition of
'ffat_fs', the current release adds the first persistent file system to the
framework. The service is located at 'libports/src/server/ffat_fs'. It uses
Genode's 'Block::Session' interface as back end. Therefore, it can be combined
with any of Genode's block-device drivers and the partition service called
'part_blk'. To see the new 'ffat_fs' service in action, please refer to the new
'libports/run/libc_ffat_fs.run' script.
On the course of our work on the 'ffat_fs' service, we enabled support for long
file names in libffat and added 'lseek' support to the 'libc_ffat' plugin.
TAR-based file-system service
=============================
The new 'tar_fs' service located at 'os/src/server/tar_fs' provides a read-only
file-system session interface by reading data from a TAR archive, which, in
turn, is fetched from a ROM service. By combining 'tar_fs' with the 'libc_fs'
plugin, it becomes easy to provide customized pseudo file systems to individual
Genode programs. For example, one instance of 'tar_fs' containing a static
website and a web-server configuration can be provided as file system to a web
server. The configuration is similar to the patterns known from the 'tar_rom'
and 'ram_fs' servers:
! <config>
! <archive name="tar_archive.tar" />
! <policy label="label_of_client" root="/rootdir/for/client" />
! </config>
The policy node allows for assigning different parts of one TAR archive to
different clients. For a practical usage example of 'tar_fs', please refer to
the 'libports/run/libc_fs_tar_fs.run' script.
Terminal improvements
=====================
Our work on running a growing number of command-line-based Unix programs via
Noux prompted us to improve our terminal implementation as needed. To ease
debugging for terminal colors, we changed the previous default color scheme to
fully saturated combinations of red, green, and blue. Albeit this looks quite
painful on the eyes, it is easier to spot wrong colors when using a program
that uses ncurses, for example Lynx. Furthermore, we added the handling of
sgr0 and sgr escape sequences and thereby enabled Lynx to become almost
usable when running within Noux.
Terminal cross-link service
===========================
The 'Terminal::Session' interface gets increasingly popular within Genode.
It is used by the UART drivers, the graphical terminal, GDB monitor, the TCP
terminal, and Noux. For most of these programs, their respective client or
server role is quite clear but we find ourselves tempted to combine components
in unusual ways. For example, to let Noux run an instance of GDB, which operates
on a terminal via a virtual character device. For remote debugging, GDB plays
the role of a terminal client and the UART driver plays the role of the server.
But when running GDB monitor on the same machine, GDB monitor will also
expect to play the role of the client. In order to connect GDB monitor
to a local instance of GDB, both of them being terminal clients, we need an
adapter component. This is where the new terminal cross-link service enters
the picture. It plays the role of a terminal server between exactly two
clients. The output of one client ends up as input to the other and vice
versa. Data sent to the server gets stored in a buffer of 4096 bytes (one
buffer per client). As long as the data to be written fits into the buffer, the
'write()' call returns immediately. If no more data fits into the buffer, the
'write()' call blocks until the other client has consumed some of the data from
the buffer via the 'read()' call. The 'read()' call never blocks. A signal
receiver can be used to block until new data is ready for reading.
The new terminal crosslink can be tested via the 'os/run/terminal_crosslink.run'
script. It is also used for the just mentioned on-target debugging scenario
demonstrated by the 'ports/run/noux_gdb.run' script.
DMA-aware and optimized packet streams
======================================
Motivated by our work on OMAP4 platform support, we introduced API extensions
for handling of DMA buffers to the following interfaces:
:'Attached_ram_dataspace':
The convenience utility for allocating and locally mapping a RAM dataspace
has been enhanced with the 'cached' constructor argument, which is true
by default. When using 'Attached_ram_dataspace' for allocating DMA buffers,
this argument should be set to false.
:Block and network packet stream:
The 'Block::Session' and 'Nic::Session' interfaces use Genode's packet stream
facility for transferring bulk payload between processes. A packet stream
combines shared memory with asynchronous notifications and thereby facilitates
the use of batched packet processing. To principally enable zero-copy semantics
for device drivers, the packet-stream buffer is now explicitly allocated as DMA
buffer. This clears the way to let the SD-card driver direct DMA transactions
right into the packet stream buffer. Consequently, when attaching the SD-card
driver directly to a file system, there is no copy of payload needed.
The 'Nic::Session' interface has further been improved by using a fast
bitmap allocator for allocations within the packet-stream buffer. This is
possible because networking packets have the MTU size as an upper limit.
In contrast to the 'Block::Session' interface where requests are relatively
large, 'Nic::Session' packets are tiny, and thus, greatly benefit from the
optimized allocator.
Libraries and applications
##########################
C runtime
=========
:File I/O:
We complemented our C runtime with support for the 'pread', 'pwrite', 'readv',
and 'writev' functions. The 'pread' and 'pwrite' functions are shortcuts for
randomly accessing different parts of a file. Under the hood, the functions are
implemented via 'lseek' and 'read/write'. To provide the atomicity of the
functions, a lock guard prevents the parallel execution of either or both
functions if called concurrently by multiple threads. The 'readv' and 'writev'
functions principally enable the chaining of multiple I/O requests.
Furthermore, we added 'ftruncate', 'poll', and basic support for (read-only)
mmapped files to the C runtime.
:Libc RPC framework headers:
Certain RPC headers of the libc are needed for compiling 'getaddrinfo.c'.
Unfortunately that means we have to generate a few header files, which we do
when we prepare the libc.
New and updated 3rd-party libraries
===================================
:Expat:
[https://expat.sourceforge.net - Expat] is an XML parsing library. The port of
this library was motivated by our goal to use the GNU debugger for on-target
debugging. GDB depends on this library.
:MPC and GMP:
We complemented our existing port of the
[https://gmplib.org - GNU multiple precision arithmetic library (libgmp)] with
support for the x86_64 and ARM architectures. This change combined with the
port of the [http://www.multiprecision.org/index.php?prog=mpc - MPC library]
enables us to build the Genode tool chain for these architectures.
:OpenSSL:
Our port of OpenSSL has been updated to version 1.0.1c. Because libcrypto
provides certain optimized assembler functions, which unfortunately are not
expressed with position-independent code, we removed this assembler code and
build libcrypto with '-DOPENSSL_NO_ASM'. Because the assembler code is not
needed anymore, its generation is also removed from 'openssl.mk'.
:Light-weight IP stack (lwIP):
We enabled the lwIP TCP/IP stack for 64-bit machines and updated the library to
version 1.4.1-rc1. With the new version, the call of 'lwip_loopback_init' is
not needed anymore because lwIP always creates a loopback device. Hence, we
will be able to remove the 'libc_lwip_loopback' in the future. For now, we keep
it around so we currently do not need to update the other targets that depend
on it.
:PCRE:
[https://www.pcre.org/ - PCRE] is a library for parsing regular rexpressions. We
require this library for our ongoing work on porting the lighttpd webserver.
Lighttpd web server
===================
The [https://www.lighttpd.net/ - Lighttpd] web server has been added to the
'ports' repository. The port runs as a native Genode application and ultimately
clears the way to hosting the genode.org website on Genode. To test drive this
scenario, please give the 'ports/run/genode_org.run' script a try.
At the current stage, the port is still quite limited. For example, it does not
make use of non-blocking sockets yet. But the 'genode_org.run' run script
already showcases very well how simple a Genode-based web-server appliance can
look like.
Device drivers
##############
OMAP4 platform drivers
======================
:HDMI output:
The new HDMI driver at 'os/src/drivers/framebuffer/omap4' implements Genode's
'Framebuffer::Session' interface by using the HDMI output of OMAP4. The current
version sets up a fixed XGA screen mode of 1024x768 with the RGB565 pixel
format.
:SD-card:
The new SD card driver at 'os/src/drivers/sd_card/omap4' allows the use of a
HDSD card with the Pandaboard as block service. The driver can be tested using
the 'os/run/sd_card.run' script. Because it implements the generic
'Block::Session' interface, it can be combined with a variety of other
components such as 'part_blk' (for accessing individual partitions) or
'ffat_fs' for accessing a VFAT file system on the SD card.
The driver uses the master DMA facility of the OMAP4 SD-card controller, which
yields to good performance at low CPU utilization. The throughput matches (and
in some cases outperforms) the Linux kernel driver. In the current version,
both modes of operation PIO and DMA are functional. However, PIO mode is
retained for benchmarking purposes only and will possibly be removed to further
simplify the driver.
:USB HID:
The OMAP4-based Pandaboard relies on USB for attaching input devices.
Therefore, we need a complete USB stack to enable the interactive use of the
board. Instead of implementing a USB driver from scratch, we built upon the USB
driver introduced with the Genode release 12.05. This driver was ported from the
Linux kernel.
:Networking:
The Pandaboard realizes network connectivity via the SMSC95xx chip attached to
the USB controller. Therefore, we enhanced our USB driver with support for USB
net and the smsc95xx driver. In addition to enabling the actual device-driver
functionality, the USB stack has received much attention concerning performance
optimizations. To speed up the allocation of SKBs, we replaced the former
AVL-tree based allocator with a fast bitmap allocator. For anonymous
allocations, we introduced a slab-based allocator. Furthermore, we introduced
the distinction between memory objects that are subjected to DMA operations
from non-DMA memory objects. The most profound conceptual optimization is the
use of transmit bursts by the driver. The Linux kernel, which our driver
originates from, does not provide an API for transmitting multiple packets as a
burst. For our driver, however, this optimization opportunity opened up thanks
to Genode's packet stream interface, which naturally facilitates the batching
of networking packets. So the driver has all the information needed to create
burst transactions.
USB driver
==========
By testing our new USB driver on a variety of real PC hardware, we discovered
several shortcomings, which we resolved. In particular, we added support for
more than one UHCI controller, make sure that the 'PIRQ' bit in the legacy
support register (PCI config space) of the UHCI controller is enabled and that
the 'Trap on IRQ' bit is disabled.
With those modifications in place, the USB driver works reliably on the tested
platforms.
Runtime environments
####################
Noux
====
Noux enables the easy reuse of unmodified GNU software on Genode by providing
a Unix-like kernel interface as user-level service. Because Noux is pivotal for
our plan to use Genode for productive work, we significantly enhanced and
complemented its feature set.
:Noux on ARM and x86_64:
For keeping the scope of the development manageable, the initial version of
Noux was tied to the x86_32 platform. This was not a principal limitation of
the approach but rather an artificial restriction to keep us focused on
functionality first. Now that Noux reaches a usable state, we desire to use it
on platforms other than x86_32. The current release enables Noux for the 64-bit
x86 and ARM architectures.
The level of support is pretty far-reaching and even includes the building and
execution of the Genode tool chain on those platforms. In the process of
enabling these platforms, we updated the Noux package for GCC to version 4.6.1,
which matches the version of the current Genode tool chain.
:Terminal file system:
Noux supports the concept of stacked file systems. The virtual file system
is defined at the start of a Noux instance driven by the static Noux
configuration. This way, arbitrary directory structures can be composed out
of file-system sessions and TAR archives. The VFS concept allows for the
easy addition of new file system types. To allow programs running in a Noux
instance to communicate over a dedicated terminal session, we added a new
file-system type that corresponds to a virtual character device node attached
to a terminal session.
:GDB running in the Noux environment:
With the terminal file system in place, we are ready to execute GDB within
Noux and let it talk to a GDB monitor instance over the terminal session
interface. From GDB's point of view, the setup looks like a remote debugging
session. But in reality both the debugging target and GDB reside in different
subtrees of the same Genode system.
:Executing shell scripts:
By inspecting the program specified to the execve system call, Noux has become
able to spawn scripts that use the '#!' syntax. If such a file is detected, it
executes the specified interpreter instead and passes the arguments specified
after the '#!' marker, followed by command-line arguments.
:Networking support:
Our work on porting various networking tools to Noux triggers us to steadily
improve the networking support introduced with Genode 12.05. In particular, we
added proper support for DNS resolving, which enables us to execute the
command-line based Lynx web browser within Noux.
:User information:
Because there are certain programs, which need the information that is stored
in 'struct passwd', we introduced configurable user information support to
Noux. One can set the user information via the '<user>' node in the Noux
config:
! <config>
! <user name="baron" uid="1" gid="1">
! <shell name="/bin/bash" />
! <home name="/home" />
! </user>
! ...
! </config>
When '<user>' is not specified, default values are used. Currently these
are 'root', 0, 0, '/bin/bash', '/'. Note that this is just a single user
implementation because each Noux instance has only one user or rather one
identity and there will be no complete multi-user support in Noux. If you need
different users, just start new Noux instances for each of them.
:New '/dev/null' and '/dev/zero' pseudo devices:
These device are mandatory for most programs (well, at least null is required
to be present for a POSIX compliant OS, which Noux is actually not). But for
proper shell-script support we will need them anyway. Under the hood, both
pseudo devices are implemented as individual file-systems and facilitate Noux's
support for stacked file systems. The following example configuration snippet
creates the pseudo devices under the '/dev' directory.
! <config>
! <fstab>
! <dir name="dev" >
! <null /> <zero />
! </dir>
! ...
! <fstab>
! ...
! </config>
Vancouver
=========
The comprehensive rework of the NOVA base platform affected the Genode version
of the Vancouver virtual machine monitor as this program used to speak directly
to the NOVA kernel. Since no kernel objects can be created outside of core
anymore, the Vancouver port had to be adjusted to solely use Genode interfaces.
L4Linux
=======
To improve the stability and performance of L4Linux on OMAP4 platforms, we
reworked parts of the Genode-specific stub drivers, in particular the
networking code. Among the improvements are the use of a high-performance
allocator for networking packets, improved IRQ safety of IPC calls (to
the Genode world), and tweaks of the TCP rmem and wmem buffer sizes to
achieve good TCP performance when running Linux with little memory.
Furthermore, we added two ready-to-use run scripts residing within
'ports-foc/run' as examples for executing L4Linux on the OMAP4-based
Pandaboard. The 'linux_panda.run' script is meant as a blue print for
experimentation. It integrates one instance of L4Linux with the native SD-card
driver, the HDMI driver, and the USB HID input driver. The
'two_linux_panda.run' script is a more elaborative example that executes two
instances of L4Linux, a block-device test, and a simple web server. Each of
the L4Linux instances accesses a different SD-card partition whereas the
block-device test operates on a third partition.

735
doc/release_notes/12-11.txt Normal file
View File

@@ -0,0 +1,735 @@
===============================================
Release notes for the Genode OS Framework 12.11
===============================================
Genode Labs
The central theme of version 12.11 of the Genode OS Framework is
self-hosting Genode on Genode. With self-hosting, we understand the execution of the
entire Genode build system within the Genode environment. There are two motivations
for pursing this line of work. First, it is a fundamental prerequisite for the
Genode developers to move towards using Genode as a day-to-day OS. Of course,
this prerequisite could be realized using one of the available virtualization
solutions. For example, we could run L4Linux on top of Genode on the Fiasco.OC
kernel and use the Genode build system from within an L4Linux instance.
However, this defeats the primary incentive behind Genode to reduce system
complexity. By having both Genode and L4Linux in the picture, we would indeed
increase the overall complexity in configuring, maintaining, and using the
system. Therefore, we would largely prefer to remove the complex Linux user
land from the picture. The second motivation is to prove that the framework and
underlying base platforms are suited and stable enough for real-world use.
If the system is not able to handle a workload like the build system,
there is little point in arguing about the added value of having a
microkernel-based system over current commodity OSes such as GNU/Linux.
We are happy to have reached the state where we can execute the unmodified
Genode build system directly on Genode running on a microkernel. As the
build system is based on GNU utilities and the GNU compiler collection,
significant effort went into the glue between those tools and the Genode API.
Section [Building Genode on Genode] provides insights into the way we achieved
the goal and the current state of affairs.
Along with the work on bringing the build system to Genode came numerous
stability improvements and optimizations all over the place, reaching from the
respective kernels, over the C runtime, the file-system implementations, memory
allocators, up to the actual programs the tool chain is composed of. Speaking
of the tool chain, the official Genode tool chain has been updated from GCC
version 4.6.1 to version 4.7.2. Thereby, all 3rd-party code packages were
subjected to testing and fixing activities.
For running the build system, the project currently focuses on NOVA and
Fiasco.OC as base platforms. However, our custom kernel platform for the ARM
architecture has also received significant improvements. With added support for
Freescale i.MX and Texas Instruments OMAP4, this platform proved to be very
well adaptable to new SoCs whereas new cache handling brings welcome
performance improvements. Furthermore, we have added experimental support for
ARM TrustZone technology, which principally enables the execution of Genode in
the so-called secure world of TrustZone while executing Linux in the so-called
normal world.
As we discovered the increasing interest in using Genode as a middleware
solution on Linux, we largely revisited the support for this kernel platform
and discovered amazing new ways to align the concept of Genode with the
mechanisms provided by the Linux kernel. Section [Linux] provides a summary
of the new approaches taken for supporting this platform.
Functionality-wise, the new version introduces support for audio drivers of
the Open Sound System, a new OMAP4 GPIO driver, improvements of the graphical
terminal, and the initial port of an SSH client.
Building Genode on Genode
#########################
On the Genode developer's way towards using Genode as a day-to-day OS, the
ability to execute the Genode build system within the Genode environment is a
pivotal step - a step that is highly challenging because the build system is
based on the tight interplay of many GNU programs. Among those
programs are GNU make, coreutils, findutils, binutils, gcc, and bash. Even
though there is a large track record of individual programs and libraries ported
to the environment, those programs used to be self-sustaining applications that
require only little interaction with other programs. In contrast, the build
system relies on many utilities working together using mechanisms such as
files, pipes, output redirection, and execve. The Genode base system does not
come with any of those mechanisms let alone the subtle semantics of the POSIX
interface as expected by those utilities. Being true to microkernel principles,
Genode's API has a far lower abstraction level and is much more rigid in scope.
To fill the gap between the requirements of the build system and the bare
Genode mechanisms, the Noux runtime environment was created. Noux is a Genode
process that acts like a Unix kernel. When started, it creates a child process,
which plays a similar role as the init process of Unix. This process communicates
via RPC messages to Noux. Using those messages, the process can perform all the
operations normally provided by a classical Unix kernel. When executed under
Noux, a process can even invoke functionalities such as fork and execve, which
would normally contradict with Genode's principles of resource management.
Over the course of the past year, more and more programs have been ported to
the Noux environment. Thereby, the semantics provided by Noux have been
successively refined so that those program behave as expected. This was an
iterative process. For example, at the beginning, Noux did not consider the
differences between 'lstat' and 'stat' as they did not matter for the first
batch of GNU programs ported to Noux. As soon as the programs got more
sophisticated, such shortcuts had to be replaced by the correct semantics. The
Genode build system is by far the most complex scenario exposed to Noux so far.
It revealed many shortcomings by both functionality implemented in Noux or the
C runtime as well as the underlying base platforms. So it proved to be a great
testing ground for analysing and improving those platform details. Therefore,
the secondary effects of self-hosting Genode on Genode in terms of stability
turned out to be extremely valuable.
The release comes with two ready-to-use run scripts for building bootable
system images that are able to execute the Genode tool chain, one for targeting
NOVA and one for targeting Fiasco.OC. Those run scripts are located at
'ports/run/' and called 'noux_tool_chain_nova.run' and 'noux_tool_chain_foc.run'
respectively. Each of those run scripts can be executed on either of those base
platforms. For example, by executing 'noux_tool_chain_nova' on Fiasco.OC, the
image will run Genode on Fiasco.OC and the tool chain will build binaries for
NOVA. When started, a build directory will be created at '/home/build'.
The Genode source code is located at '/genode'. In the '/bin' directory,
there are all the GNU programs needed to execute the tool chain. For
taking a look into the source code, 'vim' is available. To build core,
change to the build directory '/home/build' and issue 'make core'.
On Fiasco.OC, the complete Genode demo scenario can be compiled. On NOVA, the
incomplete life-time management of kernel objects will still result in an
out-of-memory error of the kernel. This kernel issue is currently being worked
on. Executing the tool chain on either of those platforms is still relatively
slow as extensive trace output is being generated and no actions have been taken to
optimize the performance so far. There are many opportunities for such
optimizations, which will be taken on as the next step.
Base framework
##############
Genode's base framework has received new support for extending session
interfaces and gained improvements with regard to interrupt handling on the x86
platform. At the API level, there are minor changes related to the CPU session
and 'Range_allocator' interfaces.
Support for specializing session interfaces
===========================================
With increasingly sophisticated application scenarios comes the desire to
extend Genode's existing session interface with new functionality. For example,
the 'Terminal::Session' interface covers plain read and write operations. It is
implemented by services such as a graphical terminal, the telnet-like TCP
terminal, or UART drivers. However, for the latter category, the breadth of the
interface is severely limited as UART drivers tend to supplement the read / write
interface with additional control functions, e.g., for setting the baud rate.
One way to go would be to extend the existing 'Terminal::Session' interface
with those control functions. However, these functions would be meaningless for
most implementations. Some of those other implementations may even desire their
own share of additions. In the longer term, this approach might successively
broaden the interface and each implementation will cover a subset only.
Because Genode aspires to keep interfaces as low-complex as possible while, at
the same time, it wants to accommodate the growing sophistication of usage
scenarios, we need a solution that scales. The solution turns out to be
strikingly simple. The RPC framework already supports the inheritance of RPC
interfaces. So it is possible to model the problem such that a new
'Uart::Session' interface derived from the existing 'Terminal::Session' will
be the host of UART-specific functionality. The only piece missing is the
propagation of both 'Uart' and 'Terminal' through the parent interface while
announcing the service. To spare the work of manually announcing the chain of
inherited interfaces from the implementor, the 'Parent::announce()' function has
been enhanced to automatically announce all service types implemented by the
announced interface. This way, a UART driver will always announce a "Uart"
and a "Terminal" service.
Improved interrupt handling
===========================
To accommodate modern x86 platforms, the session arguments of core's IRQ
service have been supplemented with the IRQ mode. There are two degrees of
freedom, namely the trigger (level / edge) and polarity (high / low). Thanks to
this addition, device drivers have become able to supply their knowledge of
devices to core.
In system scenarios with many peripherals, in particular when using the USB
driver, IRQ lines are shared between devices. Until now, Genode supported
shared interrupts for the OKL4 base platform only. To also cover the other
x86 kernels, we have generalized the interrupt sharing code and enabled this
feature on Fiasco.OC and NOVA.
Revised CPU session interface
=============================
We revisited the CPU session interface, removed no-longer used functions and
added support for assigning threads to CPUs.
The original CPU session interface contained functions for iterating through
the threads of a session. This interface was originally motivated by an
experimental statistical profiling tool that was developed at an early stage of
Genode. In the meanwhile, we discovered that the virtualization of the CPU
session interface is much more elegant to cover this use case than the
thread-iterator interface. Because the iteration has no transactional
semantics, it was unsafe to use it anyway.
To enable the use of multiple CPUs on multi-processor systems, the CPU
session interface has been enhanced with two functions, namely 'affinity'
and 'num_cpus'. The interface extension principally allows the assignment of
individual threads to CPUs. It is currently implemented on Fiasco.OC only.
On all other base platforms, 'num_cpus' returns one CPU. Note that on
the Linux platform, multiple CPUs will be used transparently.
The 'Cpu_session::state' function has been split into two functions, one
for retrieving information and one for propagating state information. The
prior interface was less explicit about the semantics of the 'state' function
as it took a non-const pointer to a 'Thread_state' object as argument.
Platform-tailored protection domains
====================================
Genode tries to provide a uniform API across all the different base platforms.
Yet, it also strives to make genuine platform features available to the
users of the framework. Examples for such features are the virtualization
support of the NOVA hypervisor or the special support for paravirtualizing
Linux on Fiasco.OC. Another example is the security model as found on the Linux
platform. Even though the security mechanisms of plain Linux are not as strong
as Genode's capability concept on a conceptual level, we still want to leverage
the available facilities such as user IDs and chroot as far as possible.
Consequently, we need a way to assign platform-specific properties to PD
sessions. With the new 'Native_pd_args' type introduced into
'base/native_types.h', there is now a way to express those platform-specific
concerns. This type is now used at all the places that deal with the creation
of protection domains such as 'Process', 'Child', and the loader.
Revised 'Range_allocator' interface
===================================
The handling of allocation errors has been refined in order to distinguish
different error conditions, in particular out-of-metadata and out-of-memory
conditions. The user of the allocator might want to handle both cases
differently. Hence we return an 'Alloc_return' value as result. In prior
versions, this type was just an enum value. With the new version, the type has
been changed to a class. This makes the differentiation of error conditions at
the caller side more robust because, in contrast to enum values, typed objects
don't get implicitly converted to bool values.
Low-level OS infrastructure
###########################
New UART session interface
==========================
To accommodate UART specific extensions of the 'Terminal::Session' interface,
in particular setting the baud rate, we introduced the new 'Uart::Session'
interface and changed the existing UART drivers to implement this
interface instead of the 'Terminal::Session' interface. Because 'Uart::Session'
inherits the 'Terminal::Session' interface, 'Uart' services announce both
"Uart" and "Terminal" at their parent.
New GPIO session interface
==========================
Embedded SoCs such as OMAP4 provide many general-purpose I/O pins, which can be
used for different purposes depending on the board where they are soldered on.
For example, the Pandaboard uses such GPIO pins to detect the presence of a
HDMI plug or control the power supply for the USB. If only one driver deals
with GPIO pins, the GPIO programming can reside in the driver. However, if
multiple drivers are used, the GPIO device resources cannot be handed out to
more than one driver. This scenario calls for the creation of a GPIO driver as
a separate component, which intermediates (and potentially multiplexes) the
access to the physical GPIO pins. The new 'Gpio::Session' interface allows one
or multiple clients to configure I/O pins, request states, as well as to
register for events happening on the pins.
Terminal
========
The graphical terminal has been enhanced with support for different built-in
font sizes and background-color handling.
In addition to those functional changes, the implementation has been decomposed
into several parts that thereby became reusable. Those parts comprise the
handling of key mappings, decoding the VT character stream, and the handling of
the character array. These functionalities are now available at
'gems/include/terminal'.
Libraries and applications
##########################
C runtime
=========
:Allocator optimized for small-object allocations:
To optimize the performance of workloads that depend on a large number of small
dynamic memory allocations, in particular the lwIP TCP/IP stack, we replaced
the memory allocator of the libc with a more sophisticated strategy. Until now,
the libc used 'Genode::Heap' as allocator. This implementation is an
AVL-tree-based best-fit allocator that is optimized for low code complexity
rather than performance for small allocations. The observation of the allocator
usage pattern of lwIP prompted us to replace the original libc malloc/free with
a version that uses slab allocators for small objects and relies on the
'Genode::Heap' for large objects only.
:Symbolic links:
Because part of our ongoing refinements of the Noux runtime is the provision of
symbolic links, support for symbolic links was added in the libc, libc plugins,
and file system servers.
lwIP
====
We updated the light-weight IP stack to version STABLE-1.4.1. Additionally,
the following optimizations were conducted to improve its performance and
robustness.
We reduced the maximum segment lifetime from one minute to one second to avoid
queuing up PCBs in TIME-WAIT state. This is the state, PCBs end up after
closing a TCP connection socket at the server side. The number of PCBs in this
state is apparently not limited by the value of 'MEMP_NUM_TCP_PCB'. One
allocation costs around 160 bytes. If clients connect to the server at a high
rate, those allocations accumulate quickly and thereby may exhaust the memory
of the server. By reducing the segment lifetime, PCBs in TIME-WAIT state are
cleaned up from the 'tcp_tw_pcbs' queue in a more timely fashion (by
'tcp_slowtmr()').
To prevent the TCP/IP stack from artificially throttling TCP throughput,
we adjusted lwIP's TCP_SND_BUF size.
From our work on optimizing the NIC stub-code performance of L4Linux as
described [https://genode.org/documentation/articles/pandaboard - here],
we learned that the use of a NIC-specific packet allocator for the
packet-stream interface is beneficial. At the lwIP back end, we still relied on
the original general-purpose allocator. Hence, we improved the lwIP back-end
code by using the bitmap-based 'Nic::Packet_allocator' allocator instead.
Standard C++ library
====================
Genode used to rely on the standard C++ library that comes with the tool chain.
However, this mechanism was prone to inconsistencies of the types defined in
the header files used at compile time of the tool chain and the types provided
by our libc. By building the C++ standard library as part of the Genode build
process, such inconsistencies cannot happen anymore. The current version of the
C++ standard library corresponds to GCC 4.7.2.
Note that the patch changes the meaning of the 'stdcxx' library for users that
happened to rely on 'stdcxx' for hybrid Linux/Genode applications. For such
uses, the original mechanism is still available, in the renamed form of
'toolchain_stdcxx'.
Device drivers
##############
Open Sound System
=================
Genode tries to re-use existing device drivers as much as possible using an
approach called device-driver environment (DDE). A DDE is a library that
emulates the environment of the original driver by translating device accesses
to the Genode API. There are many success stories of drivers successfully ported
to the framework this way. For example, using DDE-Linux, we are able to use the
Linux USB stack. Using DDE-ipxe, we are able to use iPXE networking drivers.
With Genode 12.11 we extend our arsenal of DDEs with DDE-OSS, which is a
device-driver environment for the audio drivers of the Open Sound System (OSS).
:Website of the Open Sound System:
[http://www.4front-tech.com]
The new 'dde_oss' contains all the pieces needed to use Intel HDA, AC97, and
ES1370 audio cards on Genode. On first use, the 3rd-party code can be
downloaded by issuing 'make prepare' from within the 'dde_oss' source-code
repository. Also, you need to make sure to add the 'dde_oss' repository to your
'REPOSITORIES' variable in 'etc/build.conf'.
An OSS demo configuration can be found under 'run/oss.run' and can be started
via 'make run/oss' from a Genode build directory. Be sure to adjust the
'filename' tag of the 'audio0' program. The file has to reside under
'<build-dir>/bin/'. The file format is header-less two-channel float-32 at
44100 Hz. You may use the 'sox' utility to create these audio files:
! sox -c 2 -r 44100 foo.mp3 foo.f32
OMAP4 GPIO driver
=================
The new OMAP4 GPIO driver is the first implementation of the just introduced
'Gpio::Session' interface. The driver supports two ways of interacting
with GPIO pins, by providing a static configuration, or by interacting with a
session interface at runtime. An example for a static configuration looks as
follows:
! <config>
! <gpio num="121" mode="I"/>
! <gpio num="7" mode="O" value="0"/>
! <gpio num="8" mode="O" value="0"/>
! </config>
The driver is located at 'os/src/drivers/gpio/omap4'. As reference for using
the driver, please refer to the 'os/run/gpio_drv.run' script.
Thanks to Ivan Loskutov of Ksys-Labs for contributing the session interface
and the driver!
iPXE networking drivers
=======================
We updated our device-driver environment for iPXE networking drivers to a
recent git revision and enabled support for the x86_64 architecture.
Currently, the driver covers Intel gigabit ethernet (e1000, e1000e, igb),
Intel eepro100, and Realtek 8139/8169.
Runtime environments
####################
Noux
====
The Noux runtime environment has received plenty of love thanks to the
aspiration to execute the Genode build system.
:Time:
The build system uses GNU make, which depends on time stamps of files. We do
not necessarily need a real clock. A monotonic increasing virtual time is
enough. To provide such a virtual time, the libc was enhanced with basic
support for functions like 'gettimeofday', 'clock_gettime', and 'utimes'. As
there is currently no interface to obtain the real-world time in Genode, Noux
simulates a pseudo real-time clock using a jiffies-counting thread. This
limited degree of support for time is apparently sufficient to trick tools like
ping, find, and make into working as desired.
:Improved networking support:
The Noux/net version of Noux extends the Noux runtime with the BSD-socket
interface by using the lwIP stack. This version of Noux multiplexes the
BSD-socket interface of lwIP to multiple Noux programs, each having a different
socket-descriptor name space and the principal ability to use blocking calls
such as 'select'. The code for multiplexing the lwIP stack among multiple Noux
processes has been improved to cover corner cases exposed by sophisticated
network clients, i.e., openssh.
:Directory cache for the TAR file system:
The original version of the TAR file system required a search in all TAR
records for each file lookup. This takes a long time when composing a large
directory tree out of multiple TAR archives stacked together. This is the case
for the Genode build-system scenario where we have all the files of the GNU
tools as well as the Genode source tree. Searching through thousands of records
for each call of 'stat' quickly becomes a scalability issue. Therefore, we
introduced a TAR indexing mechanism that scans each TAR file only once at the
startup of Noux and generates a tree structure representing the directory
layout. Looking up files using this index is quick.
:New packages:
With Genode-12.11, new 3rd-party packages have become available, namely
OpenSSH, the 'which' command, and all tool-chain components in their current
version. OpenSSH is still at an experimental stage. The run script at
'ports/run/noux_net_openssh_interactive.run' demonstrates how SSH can be used
to login into a remote machine.
:New pseudo file systems:
The new 'stdio' and 'random' file systems are intended to represent the pseudo
devices '/dev/random' and '/dev/tty' on Noux. Both are needed to run OpenSSH.
Note that the 'Arc4random' class, on which the random file system is based on,
currently _does not collect enough_ random bytes! It should not be used for
security-critical applications.
L4Linux
=======
The paravirtualized L4Linux kernel for the Fiasco.OC platform was updated to
SVN revision 25, which matches the Fiasco.OC SVN revision 40. We further
improved the integration of L4Linux with Genode by optimizing the stub drivers
for block devices and networking, and added principal support for running
L4Linux on SMP platforms.
Platforms
#########
NOVA
====
Genode follows the steady development of the NOVA microhypervisor very closely.
The kernel used by the framework corresponds to the current state of the master
branch of IntelLabs/NOVA.
:Improvements towards GDB support:
The NOVA-specific implementation of the CPU session interface has been improved
to accommodate the requirements posed by GDB. In particular, the 'pause',
'resume', 'state', and 'single_step' functions have been implemented. Those
functions can be used to manipulate the execution and register state of
threads. Under the hood, NOVA's 'recall' feature is used to implement these
mechanisms. By issuing a 'recall' for a given thread, the targeted thread is
forced into an exception. In the exception, the current state of the thread can
be obtained and its execution can be halted/paused.
:Maximizing contiguous virtual space:
To enable the Vancouver virtual machine monitor to hand out large amounts of
guest memory, we optimized core's virtual address space to retain large and
naturally aligned contiguous memory regions. For non-core processes, the
thread-context area that contains the stacks of Genode threads has been moved
to the end of the available virtual address space.
:Life-time management of kernel resources:
We improved the life-time management of kernel resources, in particular
capabilities, within Genode. Still the management of such kernel resources
is not on par with the Fiasco.OC version, partially because of missing
kernel functionality. This is an ongoing topic that is being worked on.
:Using the BIOS data area (BDA) to get serial I/O ports on x86:
If the I/O ports for the comport are non default (default is 0x3f8), we had to
specify manually the correct I/O ports in the source code. To avoid the need
for source-code modifications when changing test machines, we changed the core
console to read the BDA and use the first serial interface that is available.
If no serial interface is available, no device configuration will be
undertaken. The BDA can be populated via a multi-boot chain loader. Bender is
such a chain loader that can detect serial ports accessible via PCI and writes
the I/O ports to the Bios Data area (BDA). These values get then picked up by
core.
Fiasco.OC
=========
The Fiasco.OC kernel has been updated to the SVN revision 40. The update improves
SMP support and comes with various bug fixes. There is no noteworthy change
with regard to the kernel interface. We extended the number of supported
Fiasco.OC-based platforms for Genode by including the Freescale i.MX53.
To enable the use of multiple CPUs by Genode processes, the CPU session
interface has been enhanced to support configuring the affinity of threads with
CPUs. We changed the default kernel configuration for x86 and ARM to
enable SMP support and adapted L4Linux to use the new interface.
Execution on bare hardware (base-hw)
====================================
The development of our custom platform for executing Genode directly on bare
hardware with no kernel underneath went full steam ahead during the release
cycle.
:Pandaboard:
The in-kernel drivers needed to accommodate the Pandaboard, more specifically
the timer and interrupt controller, are now supported. So the Pandaboard can be
used with both 'base-hw' and 'base-foc'. Also, the higher-level platform
drivers for USB, HDMI, and SD-card that were introduced with the previous
release, are equally functional on both platforms.
:Freescale i.MX31:
We added principal support for the Freescale i.MX line of SoCs taking the
ARMv6-based i.MX31 as starting point. As of now, the degree of support is
limited to the devices needed by the kernel to operate. Pure software-based
scenarios are able to work, i.e., the nested init run script executes
successfully.
:TrustZone support:
The new VM session interface of core provides a way to execute software
in the normal world of a TrustZone system whereas Genode runs in the secure
world. From Genode's point of view, the normal world looks like a virtual
machine. Each time, the normal world produces a fault or issues a secure
monitor call, control gets transferred to the virtual machine monitor, which is
a normal user-level Genode process. The base-hw kernel has been enhanced to
perform world switches between the secure and normal world and with the ability
to handle fast interrupts (FIQs) in addition to normal interrupts. The latter
extension is needed to assign a subset of devices to either of both worlds.
Currently, the only TrustZone capable platform is the ARM CoreTile Express
CA9x4 for the Versatile Express board. For a virtual machine working properly
on top, some platform resources must be reserved. Therefore, there exist two
flavours of this platform now, one with the 'trustzone' spec-variable enabled
and one without. If 'trustzone' is specified, most platform resources (DDR-RAM,
and most IRQs) are reserved for the normal world and not available to the
secure Genode world.
:Memory attributes and caching:
We successively activated various levels of caching and improved the handling
of caching attributes propagated into the page tables. These changes resulted
in a significant boost in performance on non-emulated platforms.
Linux
=====
The Linux version of Genode was originally meant as a vehicle for rapid
development. It allows the framework components including core to be executed
as plain Linux processes. But in contrast to normal Linux programs, which
use the glibc, Genode's components interact with the kernel directly without
any C runtime other than what comes with Genode. We use the Linux version on a
regular basis to implement platform-agnostic functionality and protocols. Most
of Genode's code (except for device drivers) falls in this category. Because
the Linux version was meant as a mere tool, however, we haven't put much
thought into the principle way to implementing Genode's security concept on
this platform. Threads used to communicate over globally accessible Unix-domain
sockets and memory objects were represented as globally accessible files within
'/tmp'.
That said, even though Linux was not meant as a primary platform for Genode in
the first place, Genode can bring additional value to Linux. When considering
the implementation of a component-based system on Linux, there are several
possible approaches to take. For example, components may use DBus to
communicate, or components could pick from the manifold Unix mechanisms such as
named pipes, files, sysv-shared memory, signals, and others. Unfortunately
those mechanisms are not orthogonal and most of them live in the global name
space of the virtual file system. Whereas those mechanisms are principally able
to let processes communicate, questions about how processes get to know each
other, access-control policy, synchronization of the startup of processes are
left to the developer.
Genode, on the other hand, does provide an API for letting components
communicate but also answers those tricky questions concerning the composition
of components. This makes Genode an interesting option to build component based
applications, even on Linux. However, when used in such a context, the
limitations of the original Linux support need resolutions. Therefore, the
current release comes with a largely revised platform support for the Linux
base platform.
The changes can be summarized as follows:
:Using file descriptors as communication addresses:
Genode's synchronous RPC framework was using Unix domain sockets. Each RPC
entrypoint was represented by a pair of named files, one for sending and one
for receiving messages. In the new version, inter-process communication is
performed via file descriptors only.
:Transfer of communication rights via RPC only:
Capabilities used to be represented as a pair of the destination thread ID and
a global object ID. The thread ID has been replaced by a file descriptor that
points to the corresponding RPC entrypoint. When capabilities are transferred
as RPC arguments, those file descriptors are transferred via SCM rights
messages. This is in line with Genode's way of capability-based delegation of
access rights.
:Core-only creation of communication channels:
Communication channels used to be created locally by each process. The naming
of those channels was a mere convention. In contrast, now, communication
channels are created by core only and do not reside on the Linux virtual file
system. When creating an RPC entrypoint, core creates a socket pair and hands
out both ends to the creator of the entrypoint.
:Restricted access to memory objects:
Access to dataspace content was performed by mmap'ing a file. For a given
dataspace, the file name could be requested at core via a Linux-specific RPC
call. Now, core holds the file descriptors of all dataspaces, which are
actually unlinked files. A process that is in possession of a dataspace
capability can request the file descriptor for the content from core and mmap
the file locally. This way, access to memory objects is subjected to the
delegation of dataspace capabilities.
:Core-local process creation:
Genode used to create new processes by directly forking from the respective
Genode parent using the process library. The forking process created a PD
session at core merely for propagating the PID of the new process into core
(for later destruction). This traditional mechanism has the following
disadvantages:
First, the PID reported by the creating process to core cannot easily be
validated by core. Therefore core has to trust the PD client to not specify a
PID of an existing process, which would happen to be killed once the PD session
gets destructed. Second, there is no way for a Genode process to detect the
failure of any of its grandchildren. The immediate parent of a faulting process
could use the SIGCHLD-and-waitpid mechanism to observe its children but this
mechanism does not work transitively.
By performing the process creation exclusively within core, all Genode
processes become immediate child processes of core. Hence, core can respond to
failures of any of those processes and reflect such conditions via core's
session interfaces. Furthermore, the PID associated to a PD session is locally
known within core and cannot be forged anymore. In fact, there is actually no
need at all to make processes aware of any PIDs of other processes.
:Handling of chroot, user IDs, and group IDs:
With the move of the process creation into core, the original chroot trampoline
mechanism implemented in 'os/src/app/chroot' does not work anymore. A process
could simply escape the chroot environment by spawning a new process via core's
PD service. Therefore, chroot support has been integrated into core and the
chroot policy becomes a mandatory part of the process creation. For each process
created by core, core checks for a 'root' argument of the PD session. If a path
is present, core takes the precautions needed to execute the new process in the
specified chroot environment.
This conceptual change implies minor changes with respect to the Genode API and
the configuration of the init process. The API changes are the enhancement of
the 'Genode::Child' and 'Genode::Process' constructors to take the root path as
argument. Init supports the specification of a chroot per process by specifying
the new 'root' attribute to the '<start>' node of the process. In line with
these changes, the 'Loader::Session::start' function has been enhanced with the
additional (optional) PD argument.
In line with how the chroot path can be propagated into core, core has become
able to assign customized UIDs and GIDs to individual Genode processes or whole
Genode subsystems. The new 'base-linux/run/lx_uid.run' script contains an
example of how to use the feature.
Build system and tools
######################
The current release comes with a new tool chain based on GCC 4.7.2 and binutils
2.22. The tool-chain upgrade involved adapting the Genode code base and fixing
various issues in 3rd-party software. To obtain the new tool chain, please
refer to the tool-chain website:
:Genode tool chain:
[https://genode.org/download/tool-chain]

941
doc/release_notes/13-02.txt Normal file
View File

@@ -0,0 +1,941 @@
===============================================
Release notes for the Genode OS Framework 13.02
===============================================
Genode Labs
Traditionally, the February release of Genode is focused on platform support.
The version 13.02 follows this tradition by vastly improving Genode for the
NOVA base platform and the extending the range of ARM SoCs supported by
both our custom kernel platform and the Fiasco.OC kernel.
The NOVA-specific improvements concern three major topics, namely the added
support for running dynamic workloads on this kernel, the use of IOMMUs, and
the profound integration of the Vancouver virtual machine monitor with the
Genode environment. The latter point is particularly exciting to us because
this substantial work is the first contribution by Intel Labs to the Genode
code base. Thanks to Udo Steinberg and Markus Partheymüller for making that
possible.
Beyond the x86 architecture, the new version comes with principal support for
the ARM Cortex-A15-based Exynos 5250 SoC and the Freescale i.MX53 SoC. The
Exynos 5250 SoC has been enabled for our custom kernel as well as for the
Fiasco.OC kernel. The most significant functional improvements are a new
facility to detect faulting processes and a new mechanism for file-system
notifications.
Besides those added functionalities, the release cycle was taken as an
opportunity to revisit several aspects under the hood of the framework. A few
examples are the reworked synchronization primitives, the simplified base
library structure, the completely redesigned audio-output interface, and a
modernized timer interface.
DMA protection via IOMMU
########################
Direct memory access (DMA) of devices is universally considered as the Achilles
heel of microkernel-based operating systems. The most compelling argument in
favour of using microkernels is that by encapsulating each system component
within a dedicated user-level address space, the system as a whole becomes more
robust and secure compared to a monolithic operating-system kernel. In the
event that one component fails due to a bug or an attack, other components
remain unaffected. The prime example for such buggy components are device
drivers. By empirical evidence, those remain the most prominent trouble makers
in today's operating systems. Unfortunately however, most commodity hardware
used to render this nice argumentation moot because it left one giant loophole
open, namely bus-master DMA.
Via bus-master DMA, a device attached to the system bus is able to directly
access the RAM without involving the CPU. This mechanism is crucial for all
devices that process large amounts of data such as network adapters, disk
controllers, or USB controllers. Because those devices can issue bus requests
targeting the RAM directly and not involving the CPU altogether, such requests
are naturally not subjected by the virtual-memory mechanism implemented in the
CPU in the form of an MMU. From the device's point of view there is just
physical memory. Hence, if a driver sets up a DMA transaction, let's say a disk
driver wants to read a block from the disk, the driver tells the device about
the address and size of a physical-memory buffer where the it wants to receive
the data. If the driver lives in a user-level process, as is the case for a
microkernel-based system, it still needs to know the physical address to
program the device correctly. Unfortunately, there is nothing to prevent the
driver from specifying any physical address to the device. Consequently, a
malicious driver could misuse the device to read and manipulate all parts of
the memory, including the kernel.
[image no_iommu]
Traditional machine without IOMMU. Direct memory accesses issued by the
disk controller are not subjected to the MMU. The disk controller can
access the entity of memory present in the system.
So - does this loop hole render the micro-kernel approach useless? Of course not.
Putting each driver in a dedicated address space is still beneficial in two
ways. First, classes of bugs that are unrelated to DMA remain confined in the
driver's address space. In practice most driver issues arise from issues like
memory leaks, synchronization problems, deadlocks, flawed driver logic, wrong
state machines, or incorrect device-initialization sequences. For those classes
of problems, the microkernel argument still applies. Second, executing a driver
largely isolated from other operating-system code minimizes the attack surface
of the driver. If the driver interface is rigidly small and well-defined, it is
hard to compromise the driver by exploiting its interface.
Still the DMA issue remains to be addressed. Fortunately, modern PC hardware
has closed the bus-master-DMA loophole by incorporating a so-called IOMMU into
the system. As depicted in the following figure, the IOMMU sits between the RAM
and the system bus where the devices are attached to. So each DMA request has
to pass the IOMMU, which is not only able to arbitrate the access of DMA
requests to the RAM but also able to virtualize the address space per device.
Similar to how a MMU confines each process running on the CPU within a distinct
virtual address space, the IOMMU is able to confine each device within a
dedicated virtual address space. To tell the different devices apart, the IOMMU
uses the PCI device's bus-device-function triplet as unique identification.
[image iommu]
An IOMMU arbitrates and virtualizes DMA accesses issued by a device to the
RAM. Only if a valid IOMMU mapping exists for a given DMA access, the memory
access is performed.
Of the microkernels supported by Genode, NOVA is the first kernel that supports
the IOMMU. NOVAs interface to the IOMMU is quite elegant. The kernel simply
applies a subset of the (MMU) address space of a process (aka protection domain
in NOVA speak) to the (IOMMU) address space of a device. So the device's
address space can be managed in the same way as we normally manage the address
space of a process. The only missing link is the assignment of device address
spaces to process address spaces. This link is provided by the dedicated system
call "assign_pci" that takes a process identifier and a device identifier as
arguments. Of course, both arguments must be subjected to a security policy.
Otherwise, any process could assign any device to any other process. To enforce
security, the process identifier is a capability to the respective protection
domain and the device identifier is a virtual address where the extended PCI
configuration space of the device is mapped in the specified protection domain.
Only if a user-level device driver got access to the extended PCI configuration
space of the device, it is able to get the assignment in place.
To make NOVA's IOMMU support available to Genode programs, we enhanced the
ACPI/PCI driver with the ability to hand out the extended PCI configuration
space of a device and added a NOVA-specific extension to the PD session
interface. The new 'assign_pci' function allows the assignment of a PCI device
to the protection domain.
[image iommu_aware]
NOVAs management of the IOMMU address spaces facilities the use of
driver-local virtual addresses as DMA addresses.
Even though these mechanisms combined principally
suffice to let drivers operate with the IOMMU enabled, in practice, the
situation is a bit more complicated. Because NOVA uses the same
virtual-to-physical mappings for the device as it uses for the process, the DMA
addresses the driver needs to supply to the device must be virtual addresses
rather than physical addresses. Consequently, to be able to make a device
driver usable on systems without IOMMU as well as on systems with IOMMU, the
driver needs to be IOMMU-aware and distinguish both cases. This is an
unfortunate consequence of the otherwise elegant mechanism provided by NOVA. To
relieve the device drivers from caring about both cases, we came up with a
solution that preserves the original device interface, which expects physical
addresses. The solution comes in the form of so called device PDs. A device PD
represents the address space of a device as a Genode process. Its sole purpose
is to hold mappings of DMA buffers that are accessible by the associated
device. By using one-to-one physical-to-virtual mappings for those buffers
within the device PD, each device PD contains a subset of the physical address
space. The ACPI/PCI server performs the assignment of device PDs to PCI
devices. If a device driver intends to use DMA, it asks the ACPI/PCI driver for
a new DMA buffer. The ACPI/PCI driver allocates a RAM dataspace at core,
attaches it to the device PD using the dataspace's physical address as virtual
address, and hands out the dataspace capability to the driver. If the driver
requests the physical address of the dataspace, the returned address will be a
valid virtual address in the associated device PD. From this design follows
that a device driver must allocate DMA buffers at the ACPI/PCI server (while
specifying the PCI device the buffer is intended for) instead of using core's
RAM service to allocate buffers anonymously. The current implementation of the
ACPI/PCI server assigns all PCI devices to only one device PD. However, the
design devises a natural way to partition devices into different device PDs.
[image iommu_agnostic]
By modelling a device address space as a dedicated process (device PD),
the traditional way of programming DMA transactions can be maintained,
even with the IOMMU enabled.
Because the changed way of how DMA buffers are allocated, our existing drivers
such as the AHCI disk driver, the OSS sound driver, the iPXE network driver,
and the USB driver had to be slightly modified. We also extended DDE Kit with
the new 'dde_kit_pci_alloc_dma_buffer' function for allocating DMA buffers.
With those changes, the complete Genode user land can be used on systems with
IOMMU enabled. Hence, we switched on the IOMMU on NOVA by default.
Full virtualization on NOVA/x86
###############################
Vancouver is a x86 virtual machine monitor that is designed to run as
user-level process on top of the NOVA hypervisor. In
[https://genode.org/documentation/release-notes/11.11#Faithful_x86_PC_Virtualization_enabled_by_the_Vancouver_VMM - Genode version 11.11],
we introduced the preliminary adaptation of Vancouver to Genode. This version
was meant as a mere proof of concept, which allowed the bootup of small Guest
OSes (such as Fiasco.OC or Pistachio) inside the VMM. However, it did not
support any glue code to Genode's session interface, which limited the
usefulness of this virtualization solution at that point. We had planned to
continue the integration of Vancouver with Genode once we observed public
demand.
The move of NOVA's development to Intel Labs apparently created this demand.
It is undeniable that combining the rich user land provided by Genode with the
capabilities of the Vancouver VMM poses an attractive work load for NOVA. So
the stalled line of the integration work of Vancouver with Genode was picked up
within Intel Labs, more specifically by Markus Partheymüller. We are delighted
to be able to merge the outcome of this undertaking into the mainline Genode
development. Thanks to Intel Labs and Markus in particular for this substantial
contribution!
The features added to the new version of Vancouver for Genode are as follows:
:VMX support:
Our initial version supported AMD's SVM technology only because this was
readily supported by Qemu. With the added support for Intel VMX, Vancouver
has become able to operate on both Intel and AMD processors with hardware
virtualization support.
:Timer support:
With added support for timer interrupts, the VMM has become able to
boot a complete Linux system.
:Console support:
With this addition, the guest VM can be provided with a frame buffer and
keyboard input.
For the frame-buffer size in Vancouver, the configuration value in the
machine XML node is used. It is possible to map the corresponding memory
area directly to the guest regardless if it comes from nitpicker, a virtual
frame buffer, or the VESA driver. The guest is provided with two modes (text
mode 3 and graphics mode 0x114 (0x314 in Linux).
Pressing LWIN+END while a VM has focus resets the virtual machine. Also,
RESET and DEBUG key presses will not be forwarded to the VM anymore.
It is possible to dump a VM's state by pressing LWIN+INS keys.
The text console is able to detect idle mode, unmaps the buffer from the
guest and stops interpreting. Upon the next page fault in this area, it
resumes operation again. The code uses a simple checksum mechanism instead
of a large buffer and 'memcmp' to detect an idle text console. False
positives don't matter very much.
:Network support:
The VMM has become able to use the Intel 82576 device model from the NUL
user land to give VMs access to the network via Genode's NIC bridge service
or a NIC driver.
:Disk support:
The VMM can now assign block devices to guests using Genode's block-session
interface. The machine has to be configured to use a specified drive, which
could be theoretically routed to different partitions or services via policy
definitions. Currently the USB driver only supports one device. Genode's AHCI
driver is untested.
:Real-time clock:
By using the new RTC session interface, Vancouver is able to provide the
wall-clock time to guest OSes.
To explore the new version of the Vancouver VMM, there exists a ready-to-use
run script at 'ports/run/vancouver.run'. Only the guest OS binaries such as a
Linux kernel image and a RAM disk must be manually supplied in the
'<build-dir>/bin' directory. The run script is able to start one or multiple
instances of the VMM using the graphical launchpad.
Low-latency audio output
########################
In version 10.05, we introduced an interface for the playback of audio data
along with an audio mixer component and ALSA-based sound drivers ported from
the Linux kernel. The original 'Audio_out' session interface was based on
Genode's packet stream facility, which allows the communication of bulk data
across address spaces via a combination of shared memory and signals. Whereas
shared memory is used to transfer the payload in an efficient manner without
the need to copy data via the kernel, signals are used to manage the data flow
between the information source and sink.
[image packet_stream]
Figure [packet_stream] displays the life cycle of a packet of information
transferred from the source to the sink. The original intent behind the
packet-stream facility was the transmission of networking packets and blocks
of block devices. At the time when we first introduced the 'Audio_out'
interface, the packet stream seemed like a good fit for audio, too. However, in
the meanwhile, we came to the conclusion that this is not the case when trying
to accommodate streamed audio data and sporadic audio output at the same time.
For the output of streamed audio data, a codec typically decodes a relatively
large portion of an audio stream and submits the sample data to the mixer. The
mixer, in turn, mixes the samples of multiple sources and forwards the result
to the audio driver. Each of those components the codec, the mixer, and the
audio driver live in a separate process. By using large buffer sizes between
them, the context-switching overhead is hardly a concern. Also, the driver can
submit large buffers of sample data to the sound device without any further
intervention needed.
In contrast, sporadic sounds are used to inform the user about an immediate
event. It is ultimately expected that such sounds are played back without much
latency. Otherwise the interactive experience (e.g., of games) would suffer.
Hence, using large buffers between the audio source, the mixer, and the driver
is not an option. By using the packet stream concept, we have to settle on a
specific buffer size. A too small buffer increases CPU load caused by many
context switches and the driver, which has to feed small chunks of sample data
to the sound device. A too large buffer, however, makes sporadic sounds at low
latencies impossible. We figured out that the necessity to find a sweet spot
for picking a buffer size is a severe drawback. This observation triggered us
to replace the packet-stream-based communication mechanism of the 'Audio_out'
session interface by a new solution that we specifically designed to
accommodate both corner cases of audio output.
[image audio_out]
Similarly to the packet-stream mechanism, the new interface is based on a
combination of shared memory and signals. However, we dropped the notion of
ownership of packets. When using the packet-stream protocol depicted as above,
either the source or the sink is in charge of handling a given packet at a
given time, not both. At the points 1, 2, and 4, the packet is owned by the
source. At the points 3 and 4, the packet is owned by the sink. By putting a
packet descriptor in the submit queue or acknowledgement queue, there is a
handover of responsibility. The new interface weakens this notion of ownership
by letting the source update once submitted audio frames even after submitting
them. If there are solely continuous streams of audio arriving at the mixer,
the mixer can mix those large batches of audio samples at once and pass the
result to the driver.
[image mixer_streaming]
The mixer processes incoming data from multiple streaming sources as batches.
Now, if a sporadic sound comes in, the mixer checks the
current output position reported by the audio driver, and re-mixes those
portions that haven't been played back yet by incorporating the sporadic sound.
So the buffer consumed by the driver gets updated with new data.
[image mixer_sporadic]
A sporadic occuring sound prompts the mixer to remix packets that are
already submitted in the output queue.
Besides changing the way of how packets are populated with data, the second
major change is turning the interface into a time-triggered concept. The
driver produces periodic signals that indicate the completeness of a
played-back audio packet. This signal triggers the mixer to become active,
which in turn serves as a time base for its clients. The current playback
position is denoted alongside the sample data as a field in the memory buffer
shared between source and sink.
The new 'Audio_out' interface has the potential to align the requirements of
both streamed audio with those of sporadic sounds. As a side benefit, the now
domain-specific interface has become simpler than the original packet-stream
based solution. This becomes nowhere as evident as in the implementation of the
mixer, which has become much simpler (30% less code). The interface change
is accompanied with updates of components related to audio output, in
particular the OSS sound drivers contained in 'dde_oss', the ALSA audio driver
for Linux, the avplay media player, and the libSDL audio back-end.
Base framework
##############
Signalling API improvements
===========================
The signalling API provided by 'base/signal.h' is fairly low level. For
employing the provided mechanism by application software, we used to craft
additional glue code that translates incoming signals to C++ method
invocations. Because the pattern turned out to be not only useful but a good
practice, we added the so called 'Signal_dispatcher' class template to the
signalling API.
In addition to being a 'Signal_context', a 'Signal_dispatcher' associates a
member function with the signal context. It is intended to be used as a member
variable of the class that handles incoming signals of a certain type. The
constructor takes a pointer-to-member to the signal handling function as
argument. If a signal is received at the common signal reception code, this
function will be invoked by calling 'Signal_dispatcher_base::dispatch'. This
pattern can be observed in the implementation of RAM file system
('os/src/server/ram_fs').
Under the hood, the signalling implementation received a major improvement with
regard to the life-time management of signal contexts. Based on the observation
that 'Signal' objects are often referring to non-trivial objects derived from
'Signal_context', it is important to defer the destruction of such objects to a
point when no signal referring to the context is in flight anymore. We solved
this problem by modelling 'Signal' type as a shared pointer that operates on a
reference counter embedded in the corresponding 'Signal_context'. Based on
reference counter, the 'Signal_receiver::dissolve()' function does not return
as long as the signal context to be dissolved is still referenced by one or
more 'Signal' objects.
Trimmed and unified framework API
=================================
A though-provoking
[https://sourceforge.net/mailarchive/forum.php?thread_name=CAGQ-%3Dq27%2B_UooBiJmz9RdTE1gDmVcg9v0w-8TNgEH5fzHYiA%2BQ%40mail.gmail.com&forum_name=genode-main - posting]
on our mailing list prompted us to explore the idea to make shared libraries
and dynamically linked executables binary compatible among different kernels.
This sounds a bit crazy at first but it is not downright infeasible.
As a baby step into this direction, we unified several public headers of the
Genode API and tried to make headers private to the framework where possible.
The latter is the case for the 'base/platform_env.h' header, which is actually
not part of the generic Genode API. Hence, it was moved to the
framework-internal 'src/base/env'. Another step was the removal of
platform-specific types that are not necessarily platform-dependent. We could
remove the 'Native_lock' type without any problems. Also, we were able to unify
the IPC API, which was formerly split into the two parts 'base/ipc_generic.h'
and 'base/ipc.h' respectively. Whereas 'base/ipc_generic.h' was shared among
all platforms, the 'base/ipc.h' header used to contain platform-specific IPC
marshalling and unmarshalling code. But by moving this code from the header to
the corresponding (platform-specific) IPC library, we were able to discard the
content of 'base/ipc.h' altogether. Consequently, the former
'base/ipc_generic.h' could be renamed to 'base/ipc.h'. These changes imply no
changes at the API level.
Simplified structure of base libraries
======================================
The Genode base API used to come in the form of many small libraries, each
covering a dedicated topic. Those libraries were 'allocator_avl', 'avl_tree',
'console', 'env', 'cxx', 'elf', 'env', 'heap', 'server', 'signal', 'slab',
'thread', 'ipc', and 'lock'. The term "library" for those bits of code was
hardly justified as most of the libraries consisted of only a few .cc files.
Still the build system had to check for their inter-dependencies on each run of
the build process. Furthermore, for Genode developers, specifying the list of
base libraries in their 'target.mk' files tended to be an inconvenience. Also,
the number of libraries and their roles (core only, non-core only, shared by
both core and non-core) were not easy to capture. Hence, we simplified the way
of how those base libraries are organized. They have been reduced to the
following few libraries:
* 'cxx.mk' contains the C++ support library
* 'startup.mk' contains the startup code for normal Genode processes
On some platform, core is able to use the library as well.
* 'base-common.mk' contains the parts of the base library that are
identical by core and non-core processes.
* 'base.mk' contains the complete base API implementation for non-core
processes
Consequently, the 'LIBS' declaration in 'target.mk' files becomes simpler as
well. In the normal case, only the 'base' library must be mentioned.
New fault-detection mechanism
=============================
Until now, it was hardly possible for a parent process to respond to crashes of
child processes in a meaningful way. If a child process crashed, the parent
would normally just not take notice. Even though some special use cases such as
GDB monitor could be accommodated by the existing
'Cpu_session::exception_handler' facility, this mechanism requires the
virtualization of the 'Cpu_session interface' because an exception handler used
to refer to an individual thread rather than the whole process. For ordinary
parents, this mechanism is too cumbersome to use. However, there are several
situations where a parent process would like to actively respond to crashing
children. For example, the parent might like to restart a crashed component
automatically, or enter a special failsafe mode.
To ease the implementation of such scenarios, we enhanced the existing
'Cpu_session::exception_handler' mechanism with the provision of a
default signal handler that is used if no thread-specific handler is installed.
The default signal handler can be set by specifying an invalid thread
capability and a valid signal-context capability. So for registering a signal
handler to all threads of a process, no virtualization of the 'Cpu_session'
interface is needed anymore. The new mechanism is best illustrated by the
'os/run/failsafe.run' script, which creates a system that repeatedly spawns a
crashing child process.
Reworked synchronization primitives
===================================
We reworked the framework-internal lock interface in order to be able to use
the 'futex' syscall on the Linux base platform. Previously, the lock
implementation on Linux was based on Unix signals. In the contention case, the
applicant for a contended lock would issue a blocking system call, which gets
canceled by the occurrence of a signal. We used 'nanosleep' for this purpose.
Once the current owner of the lock releases the lock, it sends a signal to the
next applicant of the lock. Because signals are buffered by the kernel, they
are guaranteed to be received by the targeted thread. However, in situations
with much lock contention, we observed the case where the signal was delivered
just before the to-be-blocked thread could enter the 'nanosleep' syscall. In
this case, the signal was not delivered at the next entrance into the kernel
(when entering 'nanosleep') but earlier. Even though the signal handler was
invoked, we found no elegant way to handle the signal such that the subsequent
'nanosleep' call would get skipped. So we decided to leave our signal-based
solution behind and went for the mainstream 'futex' mechanism instead.
Using this mechanism required us to re-design the internal lock API, which was
originally designed with the notion of thread IDs. The 'Native_thread_id' type,
which was previously used in the lock-internal 'Applicant' class to identify a
thread to be woken up, was not suitable anymore for implementing this change.
Hence, we replaced it with the 'Thread_base*' type, which also has the positive
effect of making the public 'base/cancelable_lock.h' header file
platform-independent.
In addition to reworking the basic locking primitives, we changed the
'Object_pool' data structure to become safer to use. The former 'obj_by_*'
functions have been replaced by 'lookup_and_lock' that looks up an object and
locks it in one atomic operation. Additionally, the case that an object may
already be in destruction is handled gracefully. In this case, the lookup will
return that the object is not available anymore.
Low-level OS infrastructure
###########################
Notification mechanism for the file-system interface
====================================================
To support dynamic system scenarios, we extended Genode's file-system interface
with the ability to monitor changes of files or directories, similar to the
inotify mechanism on Linux but simpler. The new 'File_system::sigh' function
can be used to install a signal handler for an open file node. When a node is
closed after a write operation, a prior registered signal handler for this file
gets notified. Signal handlers can also be installed for directories. In this
case, the signal handler gets informed about changes of immediate nodes hosted
in the directory, i.e., the addition, renaming, or removal of nodes.
The 'ram_fs' server has been enhanced to support the new interface. So any file
or directory change can now be observed by 'ram_fs' clients.
New adapter from file-system to ROM session interface
=====================================================
The new 'fs_rom' server translates the 'File_system' session interface to the
'ROM' session' interface. Each request for a ROM file is handled by looking
up an equally named file on the file system. If no such file can be found,
then the server will monitor the file system for the creation of the
corresponding file. Furthermore, the server reflects file changes as signals
to the ROM session.
There currently exist two limitations: First, symbolic links are not handled.
Second, the server needs to allocate RAM for each requested file. The RAM is
always allocated from the RAM session of the server. Thereby, the RAM quota
consumed by the server depends on the client requests and the size of the
requested files. Therefore, one instance of the server should not be used by
untrusted clients and trusted clients at the same time. In such situations,
multiple instances of the server could be used.
The most interesting feature of the 'fs_rom' server is the propagation of
file-system changes as ROM module changes. This clears the way to use this
service to supply dynamic configurations to Genode programs.
Dynamic re-configuration of the init process
============================================
The init process has become able to respond to configuration changes by
restarting the scenario using the new configuration. To make this feature
useful in practice, init must not fail under any circumstances. Even on
conditions that were considered previously as fatal and led to the abort of
init (such as ambiguous names of the children or misconfiguration in general),
init must stay alive and responsive to configuration changes.
With this change, the init process is one of the first use cases of the dynamic
configuration feature enabled via the 'fs_rom' service and the new file-system
notifications. By supplying the configuration of an init instance via the
'fs_rom' and 'ram_fs' services, the configuration of this instance gets fetched
from a file of the 'ram_fs' service. Each time, this file is changed, for
example via VIM running within a Noux runtime environment, the init process
re-evaluates its configuration.
In addition to the support for dynamic re-configurations, we simplified the use
of conditional session routing, namely the '<if-args>' mechanism. When matching
the 'label' session argument using '<if-args>' in a routing table, we can omit
the child name prefix because it is always the same for all sessions
originating from the child anyway. By handling the matching of session labels
as a special case, the expression of label-specific routing
becomes more intuitive.
Timer interface turned into asynchronous mode of operation
==========================================================
The 'msleep' function of 'Timer::Session' interface is one of the last relics
of blocking RPC interfaces present in Genode. As we try to part away from
blocking RPC calls inside servers and as a means to unify the timer
implementation across the many different platforms supported by Genode, we
changed the interface to an asynchronous mode of operation.
Synchronous blocking RPC interfaces turned out to be constant sources of
trouble and code complexity. E.g., a timer client that also wants to respond to
non-timer events was forced to be a multi-threaded process. Now, the blocking
'msleep' call has been replaced by a mechanism for programming timeouts and
receiving wakeup signals in an asynchronous fashion. Thereby signals
originating from the timer can be handled, along with signals from other signal
sources, by a single thread. Once a timer client has registered a signal
handler using the 'Timer::sigh' function, it can program timeouts using the
functions 'trigger_once' and 'trigger_periodic', which take an amount of
microseconds as argument. For maintaining compatibility and convenience, the
interface still contains the virtual 'msleep' function. However, it is not an
RPC function anymore but a mere client-side wrapper around the 'sigh' and
'trigger_once' functions. For use cases where sleeping at the granularity of
milliseconds is too coarse (such as udelay calls by device drivers), we added
a new 'usleep' call, which takes a number of microseconds as argument.
As a nice side effect of the interface changes, the platform-specific
implementations could be vastly unified. On NOVA and Fiasco.OC, the need to use
one thread per client has vanished. As a further simplification, we changed the
timer to use the build system's library-selection mechanism instead of
providing many timer targets with different 'REQUIRES' declarations. This
reduces the noise of the build system. For all platforms, the target at
'os/src/drivers/timer' is built. The target, in turn, depends on a 'timer'
library, which is platform-specific. The various library description files are
located under 'os/lib/mk/<platform>'. The common bits are contained in
'os/lib/mk/timer.inc'.
Since the 'msleep' call is still available from the client's perspective,
the change of the timer interface does not imply an API incompatibility.
However, it provides the opportunity to simplify clients in cases that required
the maintenance of a separate thread for the sole purpose of
periodic signal generation.
Loader
======
The loader is a service that enables its clients to dynamically create Genode
subsystems. Leveraging the new fault-detection support described in section
[New fault-detection mechanism], we enabled loader clients to respond to
failures that occur inside the spawned subsystem. This is useful for scenarios
where subsystems should be automatically restarted or in situations where the
system should enter a designated failsafe mode once an unexpected fault
happens.
The loader provides this feature by installing an optional client-provided
fault handler as default CPU exception handler and a RM fault handler for all
CPU and RM sessions of the loaded subsystem. This way, the failure of any
process within the subsystem gets reflected to the loader client as a signal.
The new 'os/run/failsafe.run' test illustrate this mechanism. It covers two
cases related to the loader, which are faults produced by the immediate child
of the loader and faults produced by indirect children.
Focus events for the nitpicker GUI server
=========================================
To enable a way for applications to provide visual feedback to changed keyboard
focus, we added a new 'FOCUS' event type to the 'Input::Event' structure. To
encode whether the focus was entered or left, the former 'keycode' member is
used (value 0 for leaving, value 1 for entering). Because 'keycode' is
misleading in this context, the former 'Input::Event::keycode' function was
renamed to 'Input::Event::code'. The nitpicker GUI server has been adapted to
deliver focus events to its clients.
NIC bridge with support for static IP configuration
===================================================
NIC bridge is a service that presents one physical network adaptor as many
virtual network adaptors to its clients. Up to now, it required each client
to obtain an IP address from a DHCP server at the physical network. However,
there are situations where the use of static IPs for virtual NICs is useful.
For example, when using the NIC bridge to create a virtual network between
the lighttpd web server and the Arora web browser, both running as Genode
processes without real network connectivity.
The static IP can be configured per client of the NIC bridge using a '<policy>'
node of the configuration. For example, the following policy assigns a static
address to a client with the session label "lighttpd".
!<start name="nic_bridge">
! ...
! <config>
! <policy label="lighttpd" ip_addr="10.0.2.55"/>
! </config>
!</start>
Of course, the client needs to configure its TCP/IP stack to use the assigned
IP address. This can be done via configuration arguments examined by the
'lwip_nic_dhcp' libc plugin. For the given example, the configuration for the
lighttpd process would look as follows.
!<start name="lighttpd">
! <config>
! <interface ip_addr="10.0.2.55"
! netmask="255.255.255.0"
! gateway="10.0.2.1"/>
! </config>
!</start>
Libraries and applications
##########################
New terminal multiplexer
========================
The new 'terminal_mux' server located at 'gems/src/server/terminal_mux' is able
to provide multiple terminal sessions over one terminal-client session. The
user can switch between the different sessions using a keyboard shortcut, which
brings up an ncurses-based menu.
The terminal sessions provided by terminal_mux implement (a subset of) the
Linux terminal capabilities. By implementing those capabilities, the server
is interchangeable with the graphical terminal ('gems/src/server/terminal').
The terminal session used by the server is expected to by VT102 compliant.
This way, terminal_mux can be connected via an UART driver with terminal
programs such as minicom, which typically implement VT102 rather than the Linux
terminal capabilities.
When started, terminal_mux displays a menu with a list of currently present
terminal sessions. The first line presents status information, in particular
the label of the currently visible session. A terminal session can be selected
by using the cursor keys and pressing return. Once selected, the user is able
to interact with the corresponding terminal session. Returning to the menu is
possible at any time by pressing control-x.
For trying out the new terminal_mux component, the 'gems/run/termina_mux.run'
script sets up a system with three terminal sessions, two instances of Noux
executing VIM and a terminal_log service that shows the log output of both Noux
instances.
New ported 3rd-party libraries
==============================
To support our forthcoming port of Git to the Noux runtime environment, we
have made the following libraries available via the libports repository:
* libssh-0.5.4
* curl-7.29.0 (for now the port is x86_* only because it depends on libcrypto,
which is currently not tested on ARM)
* iconv-1.14
Device drivers
##############
Besides the changes concerning the use of IOMMUs, the following device driver
have received improvements:
:UART drivers:
The OMAP4 platform support has been extended by a new UART driver, which
enables the use of up to 4 UART interfaces. The new driver is located at
'os/src/drivers/uart/omap4'.
All UART drivers implement the 'Terminal::Session' interface, which
provides read/write functionality accompanied by a function to determine
the terminal size. The generic UART driver code shared among the various
implementations has been enhanced to support the detection of the terminal
size using a protocol of escape sequences. This feature can be enabled by
including the attribute 'detect_size="yes"' in the policy of a UART client.
This is useful for combining UART drivers with the new 'terminal_mux'
server.
:ACPI support for 64-bit machines:
In addition to IOMMU-related modifications, the ACPI driver has been enhanced
to support 64-bit machines and MCFG table parsing has been added.
:PCI support for IOMMUs:
With the added support of IOMMUs, the 'Pci::Session' interface has been
complemented with a way to obtain the extended PCI configuration space in the
form of a 'Genode::Dataspace'. Also, the interface provides a way to allocate
DMA buffers for a given PCI device. Device drivers that are meant to be used
on system with and without IOMMUs should use this interface rather than
core's RAM session interface to allocate DMA buffers.
:Real-time clock on x86:
Up to now, the x86 real-time clock driver served as a mere example for
accessing I/O ports on x86 machines but the driver did not expose any service
interface. With the newly added 'os/include/rtc_session' interface and the
added support of this interface in the RTC driver, Genode programs have now
become able to read the real-time clock. Currently, the interface is used by
the Vancouver VMM.
:USB driver restructured, support for Arndale board added:
While adding support for the Exynos-5-based Arndale board, we took the
chance to restructure the driver to improve portability to new
platforms. The most part of the driver has become a library, which is
built in a platform-specific way. The build system automatically selects
the library that fits for the platform as set up for the build directory.
Platforms
#########
NOVA
====
The NOVA base platform received major improvements that address the kernel
as well as Genode's NOVA-specific code. We pursued two goals with this line
of work. The first goal was the use of NOVA in highly dynamic settings, which
was not possible before, mainly due to lacking kernel features. The second
goal was the use of IOMMUs.
NOVA is ultimately designed for accommodating dynamic workloads on top of the
kernel. But we found that the implementation of crucial functionality was
missing. In particular, the kernel lacked the ability to destroy all kinds of
kernel objects and to reuse memory of kernel objects that had been destroyed.
Consequently, when successively creating and destroying kernel objects such as
threads and protection domains, the kernel would eventually run out of memory.
This issue became a show stopper for running the Genode tool chain on NOVA
because this scenario spawns and destroys hundreds of processes. For this
reason, we complemented the kernel with the missing functionality. This step
involved substantial changes in the kernel code. So our approach of using the
upstream kernel and applying a hand full of custom patches started to show its
limitations.
To streamline our work flow and to track the upstream kernel in a structured
way, we decided to fork NOVA's Git repository and maintain our patches in our
fork. For each upstream kernel revision that involves kernel ABI changes, we
create a separate branch called "r<number>". This branch corresponds to the
upstream kernel with our series of custom patches applied (actually rebased) on
top. This way, our additions to the upstream kernel are well documented. The
'make prepare' mechanism in the base-nova repository automates the task of
checking out the right branch. So from the Genode user's point of view, this
change is transparent.
The highly dynamic application scenarios executed on NOVA triggered several
synchronization issues in Genode's core process that had not been present on
other base platforms. The reason for those issues to occur specifically on NOVA
lies in the concurrent page fault handling as employed on this base platform.
For all classical L4-like kernels and Fiasco.OC, we use one global pager thread
to resolve all page faults that occur in the whole Genode system. In contrast,
on NOVA we use one pager thread per user thread. Consequently, proper
fine-grained synchronization between those pager threads and the other parts of
core is mandated. Even though the immediate beneficiary of these changes is the
NOVA platform, many of the improvements refer to generic code. This paves the
ground for scaling the page-fault handling on other base platforms (such as
Fiasco.OC) to multiple threads. With these improvements in place, we are able
to successfully execute the 'noux_tool_chain_nova' scenario on the NOVA kernel
and build Genode's core on NOVA. That said, however, not all issues are covered
yet. So there is still a way left to go to turn base-nova into a base platform
that is suitable for highly dynamic scenarios.
The second goal was the use of NOVA's IOMMU support on Genode. This topic is
covered in detail in section [DMA protection via IOMMU].
To be able to use and debug Genode on NOVA on modern machines that lack legacy
comports, we either use UART PCI cards or the Intel's Active Management
Technology (AMT) mechanism. In both cases, the I/O ports to access the serial
interfaces differ from the legacy comports. To avoid the need for adjusting the
I/O port base addresses per platform, we started using the chain-boot-loader
called "bender" developed by the Operating Systems Group of TU Dresden,
Germany. This boot loader is started prior the kernel, searches the PCI bus for
the first suitable device and registers the corresponding I/O port base address
at the bios data area (BDA). Genode's core, in turn, picks the I/O port base
address up from the BDA and uses the registered i8250 serial controller for its
LOG service.
Execution on bare hardware (base-hw)
====================================
The base-hw platform enables the use of Genode on ARM-based hardware without
the need for a 3rd-party kernel.
With the new release, the range of supported ARM-based hardware has been
extended to cover the following additional platforms. With the previous
release, we introduced the support for Freescale i.MX family of SoC, starting
with i.MX31. The current release adds support for the i.MX53 SoC and adds
a user-level timer driver for this platform. With the Samsung Exynos 5, the
first Cortex-A15-based SoC has entered the list of supported SoCs. Thanks to
this addition, Genode has become able to run on the
[http://www.arndaleboard.org - Howchip Arndale board]. At the current state,
core and multiple instances of init can be executed but drivers for peripherals
are largely missing. Those will be covered by our ongoing work with this SoC.
The added platforms are readily available via the 'create_builddir' tool.
To make base-hw practically usable on real hardware (i.e., the Pandaboard),
support for caches has been implemented. Furthermore, the implementation of the
signalling API underwent a redesign, which leverage the opportunities that
arise with tailoring a kernel specifically to the Genode API. As a side-benefit
of this endeavour, we could unify the 'base/signal.h' header with the generic
version and thereby took another step towards the unification of the Genode
headers across different kernels.
Microblaze platform removed
===========================
The 'base-mb' platform has been removed because it is no longer maintained.
This platform enabled Genode to run directly on the Xilinx Microblaze softcore
CPU. For supporting the Microblaze CPU architecture in the future, we might
consider integrating support for this architecture into base-hw. Currently
though, there does not seem to be any demand for it.
Fiasco.OC forked, support for Exynos 5 SoC added
================================================
In the last release cycle, we went beyond just using the Fiasco.OC kernel and
started to engage with the kernel code more intensively. To avoid that the
management of a growing number of kernel patches goes out of hand, we forked
the Fiasco.OC kernel and conduct our development in our Fiasco.OC Git
repository. When using the 'make prepare' mechanism in the 'base-foc'
repository, the new Git repository will be used automatically. There exists a
dedicated branch for each upstream SVN revision that we use. We started with
updating Fiasco.OC to the current revision 47. Hence, the current branch used
by Genode is named "r47". The branch contains the unmodified state of the
upstream SVN repository with our modifications appearing as individual commits
on top. This makes it easy to keep track of the Genode-specific modifications.
Please note that the update to Fiasco.OC requires minor adaptations inside
the 'ports-foc' repository. So for using L4Linux, "make prepare" must be
issued in both repositories 'base-foc' and 'ports-foc'.
Speaking of engaging with the kernel code, the most profound improvement is
the support for the Samsung Exynos-5-based Arndale board that we added to the
kernel. This goes hand in hand with the addition of this platform to Genode.
For creating a build directory targeting the Arndale board, just specify
"foc_arndale" to the 'create_builddir' tool. At the time of the release,
several basic scenarios including the timer driver and the USB driver are
working. Also, both Cortex-A15 CPUs of the Exynos 5 SoC are operational.
However, drivers for most of the peripherals of the Exynos-5 SoC are missing,
which limits the current scope of Genode on this platform.
Linux
=====
Since the base-linux platform became used for more than a mere development
vehicle, we are revisiting several aspects of this base platform. In the last
release, we changed the synchronous inter-process-communication mechanism to
the use of SCM rights. For the current release, it was time to have a closer
look at the memory management within core. The Linux version of core used a
part of the BSS to simulate access to physical memory. All dataspaces would
refer to a portion of 'some_mem'. So each time when core would access the
dataspace contents, it would access its local BSS. For all processes outside of
core, dataspaces were represented as files. We have now removed the distinction
between core and non-core processes. Now, core uses the same 'Rm_session_mmap'
implementation as regular processes. This way, the 'some_mem' could be
abandoned. We still use a BSS variable for allocating core-local meta data
though. The major benefit of this change is the removal of the artificial
quota restriction that was imposed by the predefined size of the 'some_mem'
array. Now, the Linux base platform can use as much memory as it likes. Because
the Linux kernel implements virtual memory, we are not bound by the physical
memory. Hence, the available quota assigned to the init process is almost
without bounds.
To implement the fault-detection mechanism described in section
[New fault-detection mechanism] on Linux, we let core catch SIGCHLD signals of
all Genode processes. If such a signal occurs, core determines the process that
produced the signal by using 'wait_pid', looks up the CPU session that belongs
to the process and delivers an exception signal to the registered exception
handler. This way, abnormal terminations of Genode processes are reflected to
the Genode API in a clean way and Genode processes become able to respond to
terminating Genode child processes.
OKL4
====
The audio stub driver has been removed from OKLinux. Because of the changed
'Audio_out::Session' interface, we needed to decide on whether to adapt the
OKLinux stub driver to the changed interface or to remove the stub driver.
Given the fact that OKLinux is not actively used, we decided for the latter.

943
doc/release_notes/13-05.txt Normal file
View File

@@ -0,0 +1,943 @@
===============================================
Release notes for the Genode OS Framework 13.05
===============================================
Genode Labs
With Genode 13.05, we have diverged quite a bit from the feature-laden plans
laid out in our [https://genode.org/about/road-map road map] as we realized
that consolidating and optimizing the current feature set will have a more
sustainable effect than functional enhancements at this point. In particular,
we addressed the problem that the ever growing diversity of platforms imposes
on the quality and coverage of testing. We also desired to extend our
systematic testing efforts to real hardware platforms, and to have a mechanism
for detecting performance regressions. Section
[Automated quality-assurance testing] details how we approached these
challenges, and how we went on analyzing Genode's network performance in
particular.
That said, we haven't completely restrained ourself from implementing new
features. Closely related to test automation but very useful in other
situations, we improved the terminal infrastructure in order to enable the
interactive use of dynamic system scenarios in headless situations. Section
[Terminal infrastructure] introduces a new command-line interface for managing
Genode subsystems.
With regard to platform support, the current release follows up on the
hardware support added in the previous releases. For Samsung Exynos-5-based
platforms, drivers for USB-3, fast-ethernet networking, gigabit networking,
eMMC, and SATA have been added. For Freescale i.MX53-based devices, new
drivers for display, touchscreen, and GPIO have become available. The
OMAP4 display driver has been enhanced to cover both LCD displays and HDMI.
Our custom base-hw kernel has been enabled on the Raspberry Pi
board. Finally, Linux/ARM was added to accompany Linux/x86 as a fully usable
Genode base platform.
Automated quality-assurance testing
###################################
One of the greatest challenges of the Genode OS Framework is preventing
regressions in the face of the growing number of supported platforms.
The challenge stems from the fact that the space of Genode scenarios grow
two-dimensional. On one axis, the software stack on top of Genode gets more
and more complex, which calls for contiguous testing. On the other axis, there
is a growing number of kernel and hardware platforms to support.
In principle, there are even more dimensions, for example the diversity
of tool chains or the diversity of the OS used on the development machine.
Luckily, the problem of tool-chain diversity could be mitigated with the
introduction of the Genode tool chain since version 11.11, which was a huge
relief. However, the mentioned two dimensions cannot be avoided. Because
manual testing of manifold scenarios of component compositions on top of many
different kernels became infeasible, we automated the task of building and
testing years ago.
The automated builder checks out the staging branch of Genode, prepares
the repositories that integrate 3rd-party code, and builds the software
for 12 different kernel/platform combinations. Not all 3rd-party software
packages are built for each combination though. But we make sure that each
piece of software is exposed to different combinations of CPU architectures
and kernels.
The build test is accompanied with automated runtime tests of various
run scripts on Qemu. Each run script listed in 'tool/autopilot.lst' is
executed on each kernel using the autopilot tool. The tests range from
stimulating low-level mechanisms (such as signal, timer, and ldso) to complex
scenarios (such as testing networking with L4Linux, or running Noux).
Both build and runtime tests are executed daily. If any of the
tests fail, the Genode developers receive a notification email.
Once all tests are passed, the staging branch can be merged into the master
branch. This way, we spare the users of Genode to deal with intermediate
problems introduced in the staging branch.
The build and runtime tests have become a fundamental tool for our
development work. With the growing variety of real hardware
(as opposed to hardware emulated via Qemu), however, our existing solution
was falling short. Even though our tests confirm that Genode is running
happily on Qemu, they won't help us to detect regressions in our device
drivers for non-Qemu hardware such as Pandaboard, Arndale, or modern PC
hardware. Furthermore, we are increasingly focussing on performance
considerations. In order to be a viable OS platform, Genode does not only need
to be able to do networking, but networking performance must be on par with
mainstream OSes. This raises the new challenge to extend our
continuous-testing tools to become continuous-benchmarking tools. The ultimate
goal is to monitor the performance of Genode on real hardware over long
periods of development.
In this release cycle, we attacked this problem in two steps. First, we
enabled Genode's run tool to target not only Qemu but real hardware, with the
premise that existing run scripts must not be changed. The second step is the
creation of new run scripts that perform benchmarks in an automated fashion.
By aggregating the results of this automatically executed benchmarks, we can
correlate performance effects with commits in our code repository.
Targeting real hardware via the run tool
========================================
In the following, we briefly describe the procedure to execute run scripts
on native hardware, for both Intel-based x86 machines and ARM-based platforms.
TFTP boot x86
~~~~~~~~~~~~~
The following description uses NOVA as an example to illustrate the usage.
Other base platforms are supported as well and can be configured analogously.
[https://os.inf.tu-dresden.de/~us15/pulsar/ - Pulsar] is a tiny boot loader
that uses PXE to fetch boot images via TFTP over the network. On the x86
architecture, Genode supports the automatic generation of Pulsar configuration
files, which can be placed directly onto a TFTP server. Genode can be booted
via Pulsar using the following steps:
* On the x86 test machine, enable "PXE boot feature" in the BIOS.
* When booting, the machine will look for a DHCP server announcing a TFTP server.
So you need to make sure to have both the DHCP server and the TFTP server
configured such that the 'pulsar' binary will be loaded as PXE binary.
* After the PXE BIOS of the test machine has loaded and started the pulsar
binary, Pulsar will look on the TFTP server for a file called
'config-XX-XX-XX-XX-XX-XX', where the sequence of 'XX' corresponds to the
MAC address of the test machine.
For example, if the MAC of the network card is 01:02:03:04:05:06, Pulsar
would request a file called 'config-01-02-03-04-05-06'.
* Using this configuration file, we direct Pulsar to the configuration
generated by the run tool. I.e., it should look as follows
! root /tftpboot/nova
! config config-00-00-00-00-00-00
The lines above tell pulsar to load another config file, which contains the
actual configuration. To instruct the run script to actually generate the
'config-00-00-00-00-00-00' file, set the following environment variables in
your shell prior executing the run script:
! export PXE_TFTP_DIR_BASE=/tftpboot
! export PXE_TFTP_DIR_OFFSET=/nova
The two-staged configuration of Pulsar may look overly complicated at first
sight but has the benefit that the run tool does not need to know the MAC
address of the test machine in order to generate the Pulsar configuration
file.
* Create a symbolic link '/tftpboot/nova' pointing to the corresponding
Genode build directory.
* The next time 'make run/printf' is invoked,
the run script will generate the 'config-00-00-00-00-00-00' in
'/tftpboot/nova'.
* When rebooting the test machine, it will load and start the printf test.
TFTP boot using U-Boot
~~~~~~~~~~~~~~~~~~~~~~
Configure your U-Boot boot loader to load the images via TFTP.
The remainder of the procedure is similar to the description for x86 above.
On ARM platforms, the run tool automatically generates the uBoot image and
creates a symbolic link into the TFTP directory.
* Pandaboard:
! export PXE_TFTP_DIR_BASE=/tftpboot
! export PXE_TFTP_DIR_OFFSET=/panda
! ln -s <genode-build-dir> /tftpboot/panda
! RUN_OPT="--target uboot" make run/printf
* Arndale board:
! export PXE_TFTP_DIR_BASE=/tftpboot
! export PXE_TFTP_DIR_OFFSET=/arndale
! ln -s <genode-build-dir> /tftpboot/panda
! RUN_OPT="--target uboot" make run/printf
Output and reset with Intel's AMT
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Most modern x86-based machines lack a COM port, which is normally used for
kernel debug messages as well as LOG messages printed by Genode's core.
However, Intel's Advanced Management Technology (AMT) can be used to obtain
the serial output of the test machine and to reset the test machine. To use
AMT with Genode's run tool, install the 'amtterm' package (version 1.3 is
known to work well) and set the following environment variables, specifying
the IP address of the test machine and the AMT password.
! export AMT_TEST_MACHINE_IP=XXX.XXX.XXX.XXX
! export AMT_TEST_MACHINE_PWD=XXXXXXXXX
Via setting the RUN_OPT environment variable, we instruct the run tool to use
AMT instead of Qemu. The following command will reset the test machine, the test
machine will load the binaries of the printf run script via PXE, and we will be
able to see the serial output of the test machine through Intel's AMT Serial
Over Line (SOL),
! RUN_OPT="--target amt" make run/printf
Output via a COM port (UART)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
If the x86 test machine, Pandaboard or Arndale test board is connected
via UART, the run tool can use a specified command to interact with it.
For example, if the UART interface of the test machine is connected directly
to the host machine at /dev/ttyUSB3, and the picocom tool is available,
the following command can be used to establish a connection:
! RUN_OPT="--target serial --serial-cmd \"picocom -b 115200 /dev/ttyUSB3\"" make run/printf
Alternatively, if the board is connected to some remote machine, which exports
the corresponding serial line via TCP/IP, the socat tool can be used for
communicating with the remote test machine:
! RUN_OPT="--target serial --serial-cmd \"socat - tcp:10.0.0.1:2000\"" make run/printf
Reset via a IP power plug NETIO-230B from Koukaam
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
At Genode Labs, we use a NETIO-230B power plug to automate power-cycling ARM
boards. This power plug can be controlled over the network. For example, if
the Pandaboard is connected to power port 3, the following command will
automatically turn on the board when the run script is started:
! RUN_OPT="--target uboot --target reset --reset-port 2 --reset-ip 10.0.0.1 --reset-user admin --reset-passwd secret" make run/printf
The '--target reset' option can be combined with '--target uboot' to
instruct the run tool to boot via TFTP (as described above) and take care
of power cycling. When the run script has finished, the specified port
will be automatically switched off by the run tool.
Of course, the IP address settings, as well as the actual user name and
password, to access the NETIO-230B power plug, have to be adjusted accordingly.
Automated benchmarking
======================
With the '--target' features added to the run tool, the road is paved to
obtain benchmark results in an automated fashion. Currently, we are most
interested in exploring the network-performance characteristics of Genode.
Network performance can be explored at different levels. We started with
looking at raw driver performance, then looked at the overhead of separating
the network application from the device driver (and thereby introducing
inter-process communication overhead), and finally explored the effects
of the TCP/IP stack.
For pursuing the packet-level performance measurements, we crafted a library
called 'net-stat', which contains the application logic of a low-level
benchmark operating at network-packet level. This library has been
successively incorporated into the 'dde_ipxe' NIC driver and the 'usb_drv'
(NIC driver via ethernet-over-USB) to measure the raw driver performance
without any microkernel overhead or TCP/IP protocol overhead.
To see the influence of the inter-process communication, namely the
packet-stream interface employed by Genode's NIC-session interface,
we implanted the same net-stat library into a NIC-session client. This
experiment enables us to compare the operation of the NIC driver
with the operation of a NIC driver separated from the NIC
application.
The raw networking tests can be executed automatically using the set
of 'network_test_nic*.run' scripts located at 'os/run'.
The scenario sends raw ethernet packets from the host machine to the
target machine. Three tests are provided:
The 'network_test_nic_raw.run' test measures the net-stat-instrumented driver
(of usb_drv and net_drv respectively) to observe the raw receive performance.
The 'network_test_nic_raw_client.run' test implements the benchmark in a
NIC-session client connected to the NIC driver running as a separate
component whereas the NIC driver is not instrumented.
The 'network_test_nic_raw_bridge_client.run' test further adds a NIC bridge
in-between the driver and the NIC-session client.
In addition to analyzing the performance on a low level, we investigated
the effects of TCP/IP for the application performance. This topic is
covered in more detail in Section [TCP/IP performance].
Terminal infrastructure
#######################
Closely related to the quality-assurance measures detailed in the previous
section, there is the arising need to interact with increasingly complex system
scenarios in headless settings. In particular when executing tests remotely on a
development board, manual user-interaction via a GUI
becomes impractical. We vastly prefer a low-bandwidth textual interface
in such situations. But how should a textual user interface for dynamic
systems comprised of many components look like? This is particularly difficult
because most development boards are equipped with merely a single UART
connector.
On a normal Genode system, the UART connector is typically used
by the kernel debugger to print debugging output, or for the interactive
use of a debugger. This leaves no interface for interacting with Genode
components. So how can we expose complex scenarios, such as concurrently
running several instances of Genode subsystems, to the user?
Our solution consists of three parts: A pseudo UART driver for Genode
that uses the kernel debugger as back end, a terminal-multiplexing
facility running on the reference platform, and a command-line based
tool for interacting with Genode. By combining those, the user
can interact with the kernel debugger, a Genode command line, and the
consoles of executed Linux instances over a single serial connection.
The pseudo UART driver called kdb_uart_drv is a Genode service that
implements the 'Uart::Session' interface. Therefore, it can be combined
with all components that use the 'Uart::Session' or the 'Terminal::Session'
interfaces, for example the Noux runtime environment, the terminal_log
service (for displaying LOG messages via the terminal interface), L4Linux, or
programs linked against the 'libc_terminal' plugin. The kdb_uart_drv component
is located at 'os/src/drivers/uart/kdb'. It does not access a real UART device
but rather uses the user-level bindings of the kernel debugger to indirectly
read and write data over the UART interface.
[image kdb_uart_drv 65%]
The kdb_uart_drv driver used for sharing one UART among the kernel
debugger, core's LOG service, and a terminal client application
running on Genode.
Figure [kdb_uart_drv] illustrates the relationship between the kernel
debugger, core's LOG service, and kdb_uart_drv. Because write operations
target the kernel debugger directly, core's LOG service gets bypassed. Output
written to the kdb_uart_drv will directly appear at the terminal program of
the host system. Because kdb_uart_drv has
direct access to the host terminal, it can leverage all facilities of the host
terminal, in particular various escape sequences for terminal manipulations.
For reading from the kernel debugger, there is no way to block for UART input.
Hence, the kdb_uart_drv periodically polls for new input with a period of 20
milliseconds. If new input is available, the driver reads as many characters as
available at once. So the runtime overhead of polling is negligible. To test
kdb_uart_drv as individual component, there is a run script provided at
'os/run/kdb_uart_drv.run'.
Thanks to kdb_uart_drv, both the kernel debugger and Genode can
share one single UART connection. So we have a principal way to let the user
interact with a Genode component that uses the 'Terminal::Session' interface.
However, typical system scenarios should accommodate not just a single program
but multiple Linux instances and native Genode applications simultaneously,
each requiring a dedicated 'Terminal::Session'. Hence, we need a way to
multiplex the 'Terminal::Session' interface between those clients. Our
multiplexing solution comes in the form of a component called terminal_mux,
which we just introduced in the
[https://genode.org/documentation/release-notes/13.02#New_terminal_multiplexer - previous release].
It uses a single terminal connection to implement a text-based user interface
to multiple virtual terminal consoles.
[image terminal_mux 40%]
Operation of the terminal_mux service.
Figure [terminal_mux] depicts the basic functioning of this component. For
terminal_mux clients, the service implements the Linux terminal capabilities.
For doing that, it shares large parts of the implementation of the existing
Genode terminal program. For each client, terminal_mux renders the client
output into a client-specific text-screen buffer. So any number of clients can
perform output on terminal_mux concurrently. According to the selection by
the user, terminal_mux periodically translates one client buffer (the
foreground buffer) to escape sequences as understood by the host terminal. This
translation is performed using the ncurses library. The user can pick the
foreground buffer using an interactive menu that can be activated via the
keyboard shortcut _Control-x_.
By combining kdb_uart_drv with terminal_mux, we created a flexible way
to let the user interact with many Genode applications. The last part
missing for a real dynamic system is a text-based command interface to
start and stop Genode subsystems. This functionality is provided by the
new cli_monitor component located at 'os/src/app/cli_monitor'.
It uses the 'Terminal::Session' interface to present a simple interactive
command line with commands for starting and stopping Genode subsystems,
entering the kernel debugger, and showing status information. It provides
tab completion and inline help to make it easily explorable. The cli_monitor
component is integrated in the scenario of the 'terminal_mux.run' script
mentioned above. Because cli_command is a 'Terminal::Session' client, it can
be interfaced with terminal_mux. This composition is illustrated by Figure
[uart_overview].
[image uart_overview 100%]
Overview of the terminal infrastructure as employed in the
demonstration scenario.
Note that in some situations, e.g., when killing subsystems, the kernel, core,
or the init process may print LOG messages. Because those messages are
naturally not routed through terminal_log, they will interfere with the
operation of terminal_mux and thereby result in visible inconsistencies.
Pressing _Control-x_ will clear such artifacts. This will bring up the
terminal_mux menu, which implicitly triggers the redraw of the entire terminal.
Base framework
##############
The current release comes with incremental improvements of the MMIO framework
API and a new utility to ease the synchronized accesses to otherwise
unsynchronized class interfaces.
:MMIO framework improvements:
For native Genode device drivers, we consistently use our
[https://genode.org/documentation/release-notes/12.02#MMIO_access_framework - MMIO framework API].
These utilities help us to safeguard the access to individual bit fields of
memory-mapped device registers and cleanly separate the declaration of device
registers from the driver logic. During the increased use of the API, we
observe that the 'Genode::Mmio' class template operates mostly on addresses
that belong to dataspaces provided by core's IO_MEM service. Those dataspaces
are typically obtained via the 'Attached_io_mem_dataspace' convenience class,
which requests the dataspace and attaches it to the local address
space at once. To further reduce repetitive code, we introduced the new
'Attached_mmio' class (located at 'os/attached_mmio.h'), which handles the
common case of making the content of a IO_MEM dataspace available through
register definitions using the 'Mmio' utility. Furthermore, the MMIO framework
API has been enhanced with a variant of the 'Mmio::wait_for()' function that
waits for whole register values rather than bits.
:Synchronized interfaces:
Most Genode programs are multi-threaded, which makes the proper use of locks
inevitable. For most data structures, Genode does not implicitly manage the
locking but expects the user of the data structures to know what he is doing.
This way, we can avoid the locking overhead if a data structure is known to be
accessed by a single thread only. If accessed by multiple threads, we usually
wrap such data structures within an accessor interface that takes care of the
locking. For example, for the 'Allocator' interface, there exists a
corresponding 'Synchronized_allocator' interface wrapper. This technique works
well as long as the number of interfaces is low -- as is the case for Genode's
base API. However, as the wrapper code is for the most part pretty dumb, we'd
like to avoid it. Also, when using the Genode API to implement programs on
top, we do not anticipate manually creating such accessor wrappers. To ease
the creation of synchronized interfaces, we introduced the new
'Synced_interface' class template. It takes a pointer to an existing interface
and a lock as arguments. An instance of a 'Synced_interface' provides
synchronized access to the wrapped interface functions via the 'operator ()'.
Because the 'Synced_interface' does not provide any means to obtain the
unsynchronized version of the interface, once wrapped, the interface cannot be
misused by subsystems that get handed over a reference to a
'Synced_interface'. To see how to employ this utility, please have a look of
how we realize the synchronization within the Vancouver VMM (in particular,
the access to the motherboard).
Low-level OS infrastructure
###########################
TCP/IP performance
==================
On the course of the automated benchmarking described in Section
[Automated quality-assurance testing], we conducted the following steps
to enable benchmarks and to improve performance at the TCP/IP level.
At application level, we desire to compare our network performance with the
performance on GNU/Linux using commodity benchmarks. For this reason, netperf
has been ported to run as native Genode using the lwIP stack. This benchmark
allows us to systematically compare our results with those achieved by Linux.
The port of netperf is available in the ports repository.
In addition to running a commodity benchmark, we pursue synthetic benchmarks
that model the behaviour of typical application scenarios, for example, a
web server that receive many small requests. This is where the added
'test-ping_client' and 'test-ping_server' tests come into play. The test
is located at 'libports/src/test/lwip/pingpong'. It is used by the
series of 'network_test_*.run' scripts located at 'libports/run'. The
run scripts exercise the test in various scenarios and thereby allow us to
systematically explore the impact of the libc and NIC bridge on the
application performance.
# Using raw lwIP without the libc
# Like the first test, but with an instance of the NIC bridge in between the
test program and the driver.
# Using lwIP with the libc socket bindings
# Like the third test, but with NIC bridge added
To keep track of the lwIP development more closely, we switched to the
Git version of lwIP instead of using a source snapshot.
Furthermore, we incorporated "window scaling" support (RFC 1323) into our
version of lwIP as we identify the TCP window size as a limiting factor
of the TCP throughput achieved via lwIP.
C runtime
=========
We added support for "resolv" functionality to the libc_lwip_nic_dhcp plugin.
Normally, a file called 'resolv.conf' is expected to be located at '/etc'.
On Genode, however, we don't have a global file system, which makes this
way of configuration cumbersome. To ease the provision of a simple default
'resolv.conf' configuration, the plugin hands out the file as a virtual file.
The configuration automatically provides the DNS server address acquired by
lwIP via DHCP. If, for some reason, this policy is not desired, the feature
can be disabled via:
! <libc resolv="no" />
*Note that the configuration of the C runtime has changed*
To foster consistency of the libc configuration, we moved the static
network "interface" attributes into the 'libc' XML node. A new configuration
of static networking would look as follows:
! <libc ip_addr="..." netmask="..." gateway="..." />
Terminal
========
Genode's custom terminal implementation has been improved to better handle
widely used escape sequences.
The new version is able to handle two-argument SGR commands with
attribute/color arguments in any order, and supports the ED, EL0, and
CUB commands.
Because the terminal classes do not rely on any 3rd-party code, they
have been moved to the os repository at 'os/include/terminal'. This way,
we can use those classes by other components of the os repository such
as the new CLI monitor.
FS-LOG service
==============
Using the new FS-LOG service residing at 'libports/os/src/server/fs_log', log
messages of different processes can be redirected to files on a file-system
service. The assignment of processes to files can be expressed in the
configuration as follows:
! <start name="fs_log">
! <resource name="RAM" quantum="2M"/>
! <provides><service name="LOG"/></provides>
! <config>
! <policy label="noux" file="/noux.log" />
! <policy label="noux ->" file="/noux_process.log" />
! </config>
! </start>
In this example, all messages originating from the noux process are directed
to the file '/noux.log'. All messages originating from children of the noux
process end up in the file '/noux_process.log'.
Liquid FB
=========
Liquid FB is a virtual framebuffer service that uses the nitpicker GUI
server as back end. The virtual framebuffer is presented as a movable
window with a title bar. Until now, we used it primarily for demonstration
purposes, i.e., it is part of Genode's default demo scenario.
Thanks to our forthcoming adaptation of Qt5 to Genode, which requires a
very similar solution to interface Qt5's platform-abstraction layer (QPA) to
Genode, liquid FB got in the spotlight of this release.
First, we took the chance to update its configuration parameters to
become more consistent with similar services such as nit_fb. As liquid_fb was
originally conceived at a time when Genode's XML parser did not support
XML attributes, its configuration syntax used to be a bit arcane. This
has changed now. Apart from this cosmetic refinement, there are two prominent
new features: Support for resizing the framebuffer window with the
mouse and support for dynamic reconfiguration of the virtual framebuffer
via Genode's configuration mechanism.
When the liquid FB window gets resized by the user, the virtual framebuffer
emits a mode-changed signal to its client, which, in turn can handle the
event by re-acquiring the frame-buffer dataspace.
The added support for dynamic reconfiguration allows for changing the
properties of a liquid FB instance via Genode's configuration mechanism.
For example, the window position and size can be manipulated this way.
Furthermore, two new configuration options have been added. The
'resize_handle' option shows or hides the resize handle widget at the
lower-right window corner (by default, it is hidden). The 'decoration' option
defines whether window decorations should be visible (default is yes). Both
options can have the values "on" or "off".
3rd-party libraries
###################
The following 3rd-party libraries have been added or updated:
* To complement libSDL, we have added ports of SDL_ttf, SDL_image,
SDL_image, SDL_mixer, and SDL_loadso. Those additions to libSDL
are used by popular libSDL-based applications such as Tuxpaint.
They are now available at the libports repository.
* GNU FriBidi 0.19.5 added to the libports repository
* Qt4 updated to version 4.8.4
* zlib updated to version 1.2.8
Device drivers
##############
Unified driver names
====================
The growing diversity of supported hardware platforms calls for improved
conventions of how to name device drivers. Otherwise, run scripts that are
meant to support a wide range of platforms will eventually become more
and more complicated due to platform-dependent conditional configuration
snippets. For example, the default framebuffer drivers of the respective
platforms used to be called "vesa_drv" (for x86), "omap4_fb_drv", or "pl11x_drv".
In order to support the different platforms, run scripts that were otherwise
platform-agnostic had to explicitly deal with those differences.
To solve this issue, we introduced a generic SPEC values for device types, for
which a default driver is expected to exist. If a platform features a
framebuffer driver, it includes the SPEC value "framebuffer". On each
platform, the default driver for the respective device has the same name. So
each of "vesa_drv", "pl11x_drv", and "omap4_fb_drv" had been renamed to
"fb_drv". This is possible because the use of those drivers is mutually
exclusive.
The same convention has been applied to GPIO drivers as well. The
corresponding SPEC value is called "gpio". The driver binaries are called
"gpio_drv".
ATAPI
=====
LBA48 support has been added to the ATAPI driver. Thanks to Ivan Loskutov!
KDB UART driver for L4/Fiasco and Fiasco.OC
===========================================
The new KDB UART driver at 'os/src/drivera/uart/kdb' uses the kernel debugger
console as backend for input and output. This is useful in the case that only
one UART is available as described in Section [Terminal infrastructure].
Examples for using the kdb_uart_drv are available in the form of the run scripts
'ports-foc/run/l4linux.run' and 'os/run/kdb_uart_drv.run'.
Revised GPIO session interface
==============================
The original design of the GPIO session interface enabled the client of a
single session to interact with any number GPIO pins. Each function of the
interface took a GPIO number as first argument, which addressed the GPIO pin.
To simplify the interface and to enable fine-grained GPIO-assignment policies,
the interface has been changed to provide access to a single GPIO pin per
session only. At session creation time, the client specifies a single GPIO
pin, to which the session refers. This information can be evaluated for the
session routing. So access-control policies can be easily implemented per GPIO
pin. The server stores the pin as part of the session context and implicitly
uses the pin for operations on the session interface.
Furthermore, a generic driver interface for GPIO-class-device drivers
has been introduced. The new interface at 'os/include/gpio' alleviates the
need to implement the boilerplate code to interface the driver with Genode.
The existing GPIO drivers for OMAP4 and i.MX53 are the first beneficiaries of
these changes.
Exynos 5 SoC
============
After principally enabling the Exynos 5 SoC platform in the previous
release, we moved on with extending the device-driver coverage of this SoC. In
particular, we addressed USB networking, XHCI (USB-3), Gigabit networking over
USB-3, eMMC, and SATA.
The development of those device drivers follows our rationale that guided our
[https://genode.org/documentation/articles/pandaboard - previous work on the OMAP4 platform].
For the USB driver, we employed the device-driver-environment (DDE) approach
for reusing the Linux USB stack and the host controller drivers. In contrast,
the eMMC and SATA drivers are built as genuine Genode drivers with no
3rd-party code used.
Technically, the addition of Exynos-5 support to our USB driver was
an evolutionary step. It required us to add the corresponding EHCI
controller and to supply a few additions to the device-driver
environment. To simplify the driver, we decided to let the driver
rely on the platform initialization as performed by the U-Boot boot
loader. Since the initialization is performed during the boot process
already, there is no need to do this work twice. Because the platforms
supported by the USB driver become more and more diverse, we re-organized the
internal structure of the 'dde_linux' repository to keep those platforms well
separated. Furthermore, we reworked the memory management of the USB driver to
improve the utilization of the available RAM. The new solution employs Genode's
concept of managed dataspaces to manage a part of the local address-space
layout manually. This helps us to implement a fast translation of driver-local
virtual addresses to physical addresses as needed for issuing DMA requests.
The eMMC driver builds upon our protocol implementation for the SD-card
protocol, which was originally developed for the OMAP4 SD-card driver.
Because we kept the SD-card protocol implementation well separated
from the host-controller driver, it was possible to leverage parts of our
existing work for the eMMC driver. Because the eMMC protocol is an extension
of the SD-card protocol, however, we needed to enhance the protocol
implementation accordingly. The extension comprises support for the
MMC_SEND_EXT_CSD, MMC_SEND_OP_COND, and STOP_TRANSMISSION commands as well as
the MMC detection. The host controller driver was implemented from scratch
with the help of I/O access traces gathered from instrumenting the U-Boot boot
loader and the Linux kernel. The driver operates the eMMC in high-speed, 8-bit
mode at 52 MHz using DMA. The implementation can be found at
'os/src/drivers/sd_card/exynos5'.
The initial version of our new SATA driver for Exynos 5 has been implemented
from the ground up. Even though it is at an early stage, it has been
successfully tested with a UDMA-133 disk, e.g., our generic block test
is passed and the disk can be attached as a block device to an instance of
L4Linux.
Freescale i.MX SoC
==================
The support for the Freescale i.MX53 SoC has been extended by a number of
devices. All drivers reside in the os repository under the 'os/src/drivers'
subdirectory.
The general-purpose I/O (GPIO) driver located at 'gpio/imx53' implements the
revised GPIO-session interface.
The i.MX53 input driver provides support for the input devices featured on the
i.MX53 SABRE tablet. The tablet uses an Egalaxy touchscreen and Freescale's
MPR121 capacitative touch buttons. Both are supported by the new driver. The
driver is located at 'input/imx53'.
The new framebuffer driver for the i.MX53 quick-start board (QSB) as well as
the SABRE tablet comes with special support for using the
hardware overlay feature provided by the i.MX53 image processing unit (IPU)
Access to the overlay is implemented via an IPU-specific extension
of the framebuffer-session interface. To combine the driver well with
nitpicker using alpha-channels, optional support for double-buffering
is provided. The driver is located at 'framebuffer/imx53'.
As an abstraction of platform features that need to be accessed by
multiple drivers, a so-called platform driver has been introduced.
The platform driver safeguards the access to global resources such
as clocks and system-configuration bits. It can be found at 'platform/imx53'.
OMAP4 SoC
=========
The OMAP4 framebuffer driver used to support HDMI only, which was used
for connecting a display to the Pandaboard. To make the driver usable on
phones and tablets, the driver has been enhanced to support LCD output. Thanks
to Alexander Tarasikov for the patch and the insightful story about
[https://allsoftwaresucks.blogspot.com/2013/05/porting-genode-to-commercial-hardware.html - porting Genode to the B&N Nook HD+ tablet]!
USB
===
The USB driver of the 'dde_linux' repository has received substantial
improvements both feature-wise and under the hood.
First and foremost, the Linux device-driver environment, on which the
driver is based on, has been updated from kernel version 3.2 to version
3.9 as the latter version includes drivers for recent host controllers
such as DWC3 out of the box.
DWC3 is the host controller employed on the Exynos-5-based
Arndale platform for USB 3. We added the support needed to operate this
controller in XHCI mode and added support for Gigabit networking through
the ASIX AX88179 Gigabit-Ethernet Adapter as well as USB storage support.
Apart from extending the device-driver coverage, we revised the driver
internally. The back-end allocators for DMA buffers and normal memory have been
rewritten to allocate RAM more sparingly. Furthermore, we enabled the USB
driver for 64-bit x86 machines and improved the support for HID keyboards,
including the application of quirks to cherry keyboards.
*Note the change of the USB configuration*
With the addition of XHCI, the USB driver supports a growing number
of host controllers. In some situations, it is desirable to constrain the
driver to a subset of controllers only. For example, on the Arndale platform,
we desire to use a dedicated USB stack for XHCI, which operates completely
independent from the USB stack accessing USB-2. This way, gigabit networking
over USB-3 won't interfere with the operation of USB-2. To make this
possible, we added new configuration options to the USB driver.
With the new scheme, host controllers must be explicitly enabled in the
configuration. Supported config attributes are: 'uhci', 'ehci', and 'xhci'.
For example, a configuration snippet to enable UHCI and EHCI looks as
follows:
! <config uhci="yes" ehci="yes">
Updated iPXE device-driver environment
======================================
The iPXE device-driver environment was update to the most recent
iPXE upstream Git version in order to benefit from upstream improvements
of the Intel E1000 NIC driver.
Runtime environments
####################
Vancouver VMM on NOVA
=====================
Vancouver is the user-level virtual-machine monitor that accompanies the
NOVA hypervisor for hosting unmodified guest operating systems.
The most active line of development is led by Julian Stecklina at TU Dresden
via a fork called Seoul. In contrast to the original version of Vancouver,
this fork is open for outside contributions. Hence, it represents an ideal
platform for those parties with a stake in Vancouver to collaborate, i.e.,
the NUL userland, the NOVA runtime environment of TUD, and Genode.
In the current state of the transition, the Hip structure from Genode
is reused. String functions, which were formerly taken from NUL are now
provided by a stripped-down version of the C library called
'seoul_libc_support'. The nul/config.h is replaced by just using a constant
value in the one place where the file was needed.
The Genode-specific back ends of Vancouver, as largely introduced with the
previous Genode release, have been improved in several respects:
* CPUID 0x40000000: This instruction is issued by Linux when the KVM
guest support is compiled in. We have to return deterministic values to let
the Linux kernel survive.
* Replaced busy thread startup synchronization by proper locking.
* New locking scheme: We replaced the error-prone manual locking with the
use of the freshly introduced 'Synced_interface' for the motherboard and the
VCPU dispatcher. Also, all globally visible locks have been removed. They are
explicitly passed to subsystems only when needed.
* Improved PS/2 mouse back-end:
The previous version of the PS/2 mouse back end managed mouse-motion
events in a strange way, effectively throwing away most information
about the motion vector. Furthermore, the tracking of the mouse-button
states were missing. So drag-and-drop in a guest OS won't work. The new
version fixes those issues. For the transformation of input events to
PS/2 packets, the 'Genode::Register' facility is used, which greatly
simplifies the code.
L4Linux on Fiasco.OC
====================
We improved the memory management of L4Linux on Genode in two ways.
The first improvement is concerned about the upper limit of memory per Linux
instance. The corresponding discussion can be found at
[https://github.com/genodelabs/genode/issues/414 - issue #414].
We changed our L4Re emulation library to match the semantics of the original
L4Re more closely. Furthermore, we removed a heuristic in the L4Linux kernel,
which assumed that all kernel-local addresses above 0x8000000 refer to device
resources. In our version of L4Linux, there exist no MMIO resources. In
contrary, the virtual addresses above this addresses are used for normal
memory. By removing this artificial restriction with regard to the virtual
memory layout of the L4Linux kernel, we can host a larger kernel memory area.
The second improvement is concerned with the allocation of L4Linux
memory at Genode's core. Until now, L4Linux used to allocate its memory
as one contiguous RAM dataspace at core's RAM service. Core tries to
naturally align the allocation to improve the likelihood for large-page
mappings. So a dataspace is likely to be physically located at a
power-of-two boundary larger or equal than the dataspace size. For example,
the allocation of a 100 MiB RAM dataspace for a Linux instance will
be located at a 128 MiB boundary. If multiple of such allocations happen
sub-sequentially, this allocation strategy results in 28 MiB gaps between
100 MiB dataspaces. This memory cannot be used for large contiguous
allocations anymore. So even if the available memory capacity is far
larger than 100 MiB, an allocation of a 100 MiB block may fail.
To relieve this problem, we weakened the requirement for contiguous memory
by assembling L4Linux memory from multiple chunks of small dataspaces.
For example, by using a chunk size of 16 MiB, core's best-fit allocator
will have a better chance to find a more suited position for allocation
when aligning the block to a 16 MiB boundary compared to the allocation
of a larger block. Furthermore, slack memory can be used more efficiently
because smaller gaps (such as a 20 MiB gap) remain to be usable for L4Linux.
The discussion of this topic and the individual patch can be found at
[https://github.com/genodelabs/genode/issues/695 - issue #695].
Furthermore, the L4Linux block driver has been improved to support large
partitions.
Platforms
#########
Execution on bare hardware (base-hw)
====================================
Raspberry Pi
~~~~~~~~~~~~
Principal support for the Raspberry Pi platform has been added to the base-hw
kernel. The popular Raspberry Pi board is based on an ARMv6 Broadcom BCM2835
SoC. The current scope of the platform support comprises:
* IRQ controller driver: Because the interrupt controller uses a cascade of
registers, we settled on the following IRQ enumeration scheme.
IRQ numbers 0..7 refer to the basic IRQs.
IRQ numbers 8..39 refer to GPU IRQs 0..31.
IRQ numbers 40..71 refer to GPU IRQs 32..63.
* The kernel employs the so-called system timer for the preemptive scheduling.
* Core's LOG messages are printed over the PL011-based UART.
* The user-level timer driver uses the so-called ARM timer, which is a
slightly modified SP804 timer device.
Up to this point, a few device driver are missing to use Genode on the
Raspberry Pi in practice, most notably USB.
To build and run Genode on the Raspberry Pi, create a new build directory
via the 'create_builddir' tool, specifying 'hw_rpi' as platform.
User-level timer driver for Arndale platform
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
By adding our new Exynos 5250 PWM timer driver, the base-hw kernel can now
be used for executing meaningful scenarios on the Arndale board including
the USB stack and networking.
Linux
=====
Until now, Genode on Linux supported x86-based platforms only.
The newly added 'linux_arm' platform clears the way to run Genode directly on
Linux-based ARM platforms. Genode's entire software stack is supported,
including the dynamic linker, graphical applications, and Qt4.
As a known limitations, the libc 'setjmp()'/'longjmp()' doesn't currently
save/restore floating point registers.
Build system and tools
######################
The run tool has been enhanced as detailed in Section
[Automated quality-assurance testing].

995
doc/release_notes/13-08.txt Normal file
View File

@@ -0,0 +1,995 @@
===============================================
Release notes for the Genode OS Framework 13.08
===============================================
Genode Labs
The release of version 13.08 marks the 5th anniversary of the Genode OS
framework. We celebrate this anniversary with the addition of three major
features that we have much longed for, namely the port of Qt5 to Genode,
profound multi-processor support, and a light-weight event tracing
framework. Additionally, the new version comes with new device drivers for
SATA 3.0 and power management for the Exynos-5 SoC, improved virtualization
support on NOVA on x86, updated kernels, and integrity checks for
downloaded 3rd-party source code.
Over the course of the past five years, Genode's development was primarily
motivated by adding and cultivating features to make the framework fit for as
many application areas as possible. Now that we have a critical mass of
features, the focus on mere functionality does not suffice anymore. The
question of what Genode can do ultimately turns into the question of how well
Genode can do something: How stable is a certain workload? How does networking
perform? How does it scale to multi-processor systems? Because we are lacking
concise answers to these kind of questions, we have to investigate.
When talking about stability, our recently introduced automated testing
infrastructure makes us more confident than ever. Each night, over 200
automated tests are performed, covering various kernels and several hardware
platforms. All those tests are publicly available in the form of so-called run
scripts and are under continues development.
Regarding performance investigations, recently we have begun to benchmark
application performance focusing on network throughput. Interestingly, our
measurements reveal significant differences between the used kernels, but
also shortcomings in our software stack. For example, currently we see that
our version of lwIP performs poorly with gigabit networking. To thoroughly
investigate such performance issues, the current version adds support
for tracing the behaviour of Genode components. This will allow us to get a
profound understanding of all inter-component interaction that are on the
critical path for the performance of complex application-level workloads.
Thanks to the Genode architecture, we could come up with a strikingly simple,
yet powerful design for a tracing facility. Section
[Light-weight event tracing] explains how it works.
When it comes to multi-processor scalability, we used to shy away from such
inquiries because, honestly, we haven't paid much consideration to it. This
view has changed by now. With the current release, we implemented the
management of CPU affinities right into the heart of the framework, i.e.,
Genode's session concept. Additionally, we cracked a damn hard nut by enabling
Genode to use multiple CPUs on the NOVA hypervisor. This kernel is by far the
most advanced Open-Source microkernel for the x86 architecture. However,
NOVA's MP model seemed to inherently contradict with the API design of Genode.
Fortunately, we found a fairly elegant way to go forward and we're able to
tame the beast. Section [Enhanced multi-processor support] goes into more
detail.
Functionality-wise, we always considered the availability of Qt on Genode as a
big asset. With the current release, we are happy to announce that we finally
made the switch from Qt4 to Qt5. Section [Qt5 available on all kernels] gives
insights into the challenges that we faced during porting work.
In addition to those highlights, the new version comes with improvements all
over the place. To name a few, there are improved support for POSIX threads,
updated device drivers, an updated version of the Fiasco.OC kernel and
L4Linux, and new device drivers for Exynos-5. Finally, the problem of
verifying the integrity of downloaded 3rd-party source codes has been
addressed.
Qt5 available on all kernels
############################
Since its integration with Genode version 9.02, Qt4 is regarded as
one of the most prominent features of the framework. For users, combining Qt
with Genode makes the world of sophisticated GUI-based end-user applications
available on various microkernels. For Genode developers, Qt represents by far
the most complex work load natively executed on top of the framework API,
thereby stressing the underlying system in any way imaginable. We have been
keeping an eye on Qt version 5 for a while and highly anticipate the direction
where Qt is heading. We think that the time is right to leave Qt4 behind to
embrace Qt5 for Genode.
For the time being, both Qt4 and Qt5 are available for Genode, but Qt4 is
declared as deprecated and will be removed with the upcoming version 13.11.
Since Qt5 is almost API compatible to Qt4, the migration path is relatively
smooth. So we recommend to move your applications over to Qt5 during the
next release cycle.
In the following, we briefly describe the challenges we faced while adding Qt5
support to Genode, point you to the place where to find Qt5 in the source
tree, and give quick-start instructions for getting a Qt5 application
scenario running.
We found that the biggest architectural difference between version 4 and
version 5 is the use of the so-called Qt Platform Abstraction (QPA) interface,
which replaces the former Qt Window System (QWS).
Moving from QWS to QPA
======================
With Qt4, we relied on QWS
to perform the window handling. A Qt4 application used to create a session to
Genode's GUI server (called nitpicker) and applied its graphical output onto a
virtual framebuffer. The virtual framebuffer was not visible per se. To make
portions of the virtual framebuffer visible on screen, the application had to
create so-called nitpicker views. A view is a rectangular area of the physical
screen that displays a (portion of) the virtual framebuffer. The position,
size, and stacking order of views is managed by the application. For each Qt
window, the application would simply create a corresponding nitpicker view and
maintain the consistency of the view with the geometry of the Qt window. Even
though each Qt application seemingly operated as a full-screen application
with the windows managed by the application-local QWS, the use of nitpicker
views still allowed the integration of any number of Qt applications into one
windowed environment.
With the advent of compositing window managers, the typical way of how an
application interacts with the window system of the OS changed. Whereas
old-generation GUIs relied on a tight interplay of the application with the
window server in order to re-generate newly exposed window regions whenever
needed (e.g., revealing the window content previously covered by another
window), the modern model of a GUI server keeps all pixels of all windows in
memory regardless of whether the window is visible or covered by other
windows. The use of one pixel buffer per window seems wasteful with respect to
memory usage when many large windows are overlapping each other. On the other
hand, this technique largely simplifies GUI servers and makes the
implementation of fancy effects, like translucent windows, straight forward.
Since memory is cheap, the Qt developers abandoned the old method and fully
embraced the buffer-per-window approach by the means of QPA.
For Genode, we faced the challenge that we don't have a window server
in the usual sense. With nitpicker, we have a GUI server, but with a more
radical design. In particular, nitpicker leaves the management of window
geometries and stacking to the client. In contrast, QPA expects the window
system to provide both means for a user to interactively change the window
layout and a way for an application to define the properties (such as the
geometry, title, and visibility) of its windows.
The obviously missing piece was the software component that deals with window
controls. Fortunately, we already have a bunch of native nitpicker applications
that come with client-side window controls, in particular the so-called liquid
framebuffer (liquid_fb). This nitpicker client presents a virtual framebuffer
in form of a proper window on screen and, in turn, provides a framebuffer and
input service. These services can be used by other Genode processes, for
example, another nested instance of nitpicker.
This way, liquid_fb lends itself to be the interface between the nitpicker
GUI server and QPA.
For each QPA window, the application creates a new liquid_fb instance as a
child process. The liquid_fb instance will request a dedicated nitpicker
session, which gets routed through the application towards the parent of the
application, which eventually routes the request to the nitpicker GUI server.
Finally, the liquid_fb instance announces its input and framebuffer services
to its parent, which happens to be the application. Now, the application is
able to use those services in order to access the window. Because the
liquid_fb instances are children of the application, the application can
impose control over those. In particular, it can update the liquid_fb
configuration including the window geometry and title at any time. Thanks to
Genode's dynamic reconfiguration mechanism, the liquid_fb instances are able
to promptly respond to such reconfigurations.
Combined, those mechanisms give the application a way to receive user input
(via the input services provided by the liquid_fb instances), perform
graphical output (via the virtual framebuffers provided by the liquid_fb
instances), and define window properties (by dynamically changing the
respective liquid_fb configurations). At the same time, the user can use
liquid_fb's window controls to move, stack, and resize application windows as
expected.
[image qt5_screenshot]
Steps of porting Qt5
====================
Besides the switch to QPA, the second major change was related to the build
system. For the porting work, we use a Linux host system to obtain the
starting point for the build rules. The Qt4 build system would initially
generate all Makefiles, which could be inspected and processed at once. In
contrast, Qt5 generates Makefiles during the build process whenever needed.
When having configured Qt for Genode, however, the build on Linux will
ultimately fail. So the much-desired intermediate Makefiles won't be created.
The solution was to have 'configure' invoke 'qmake -r' instead of 'qmake'.
This way, qmake project files will be processed recursively. A few additional
tweaks were needed to avoid qmake from backing out because of missing
dependencies (qt5_configuration.patch). To enable the build of the Qt tools
out of tree, qmake-specific files had to be slightly adapted
(qt5_tools.patch). Furthermore, qtwebkit turned out to use code-generation
tools quite extensively during the build process. On Genode, we perform this
step during the 'make prepare' phase when downloading and integrating the Qt
source code with the Genode source tree.
For building Qt5 on Genode, we hit two problems. First, qtwebkit depends on
the ICU (International Components for Unicode) library, which was promptly
ported and can be found in the libports repository. Second, qtwebkit
apparently dropped the support of the 'QThread' API in favor of
POSIX-thread support only. For this reason, we had to extend the coverage
of Genode's pthread library to fulfill the needs of qtwebkit.
Once built, we entered the territory of debugging problems at runtime.
* We hit a memory-corruption problem caused by an assumption of 'QArrayData'
with regard to the alignment of memory allocated via malloc. As a
work-around, we weakened the assumptions to 4-byte alignment
(qt5_qarraydata.patch).
* Page faults in QWidgetAnimator caused by
use-after-free problems. Those could be alleviated by adding pointer
checks (qt5_qwidgetanimator.patch).
* Page faults caused by the slot function 'QWidgetWindow::updateObjectName()'
with a 'this' pointer of an incompatible type 'QDesktopWidget*'.
As a workaround, we avoid this condition by delegating the
'QWidgetWindow::event()' that happened to trigger the slot method to
'QWindow' (base class of 'QWidgetWindow') rather than to a
'QDesktopWidget' member (qt5_qwidgetwindow.patch).
* We observed that Arora presented web sites incomplete, or including HTTP
headers. During the evaluation of HTTP data, a signal was sent to another
thread, which activated a "user provided download buffer" for optimization
purposes. On Linux, the receiving thread was immediately scheduled and
everything went fine. However, on some kernels used by Genode, scheduling
is different, so that the original thread continued to execute a bit longer,
ultimately triggering a race condition. As a workaround, we disabled the
"user provided download buffer" optimization.
* Page faults in the JavaScript engine of Webkit. The JavaScript
'RegExp.exec()' function returned invalid string objects. We worked around
this issue by deactivating the JIT compiler for the processing of
regular expressions (ENABLE_YARR_JIT).
The current state of the Qt5 port is fairly complete. It covers the core, gui,
jscore, network, script, scriptclassic, sql, ui, webcore, webkit, widgets,
wtf, and xml modules. That said, there are a few known limitations and
differences compared to Qt4. First, the use of one liquid_fb instance per
window consumes more memory compared to the use of QWS in Qt4. Furthermore,
external window movements are not recognized by our QPA implementation yet.
This can cause popup menus to appear at unexpected positions. Key repeat is
not yet handled. The 'QNitpickerViewWidget' is not yet adapted to Qt5. For this
reason, qt_avplay is not working yet.
Test drive
==========
Unlike Qt4, which was hosted in the dedicated 'qt4' repository, Qt5 is
integrated in the libports repository. It can be downloaded and integrated
into the Genode build system by issuing 'make prepare' from within the
libports repository. The Qt5 versions of the known Qt examples are located at
libports/src/app/qt5. Ready-to-use run scripts for those examples are available
at libports/run.
Migration away from Qt4 to Qt5
==============================
The support for Qt4 for Genode has been declared as deprecated. By default,
it's use is inhibited to avoid name aliasing problems between both versions.
Any attempt to build a qt4-based target will result in a message:
!Skip target app/qt_launchpad because it requires qt4_deprecated
To re-enable the use of Qt4, the SPEC value qt4_deprecated must be defined
manually for the build directory:
!echo "SPECS += qt4_deprecated" >> etc/specs.conf
We will keep the qt4 repository in the source tree during the current
release cycle. It will be removed with version 13.11.
Light-weight event tracing
##########################
With Genode application scenarios getting increasingly sophisticated,
the need for thorough performance analysis has come into spotlight.
Such scenarios entail the interaction of many components.
For example, with our recent work on optimizing network performance, we
have to consider several possible attack points:
* Device driver: Is the device operating in the proper mode? Are there
CPU-intensive operations such as allocations within the critical path?
* Interface to the device driver: How frequent are context switches between
client and device driver? Is the interface designed appropriately for
the access patterns?
* TCP/IP stack: How does the data flow from the raw packet level to the
socket level? How dominant are synchronization costs between the involved
threads? Are there costly in-band operations performed, e.g., dynamic
memory allocations per packet?
* C runtime: How does integration of the TCP/IP stack with the C runtime
work, for example how does the socket API interfere with timeout
handling during 'select' calls?
* Networking application
* Timer server: How often is the timer consulted by the involved components?
What is the granularity of timeouts and thereby the associated costs for
handling them?
* Interaction with core: What is the profile of the component's interaction
with core's low-level services?
This example is just an illustration. Most real-world performance-critical
scenarios have a similar or even larger scope. With our traditional
tools, it is hardly possible to gather a holistic view of the scenario. Hence,
finding performance bottlenecks tends to be a series of hit-and-miss
experiments, which is a tiresome and costly endeavour.
To overcome this situation, we need the ability to gather traces of component
interactions. Therefore, we started investigating the design of a tracing
facility for Genode one year ago while posing the following requirements:
* Negligible impact on the performance, no side effects:
For example, performing a system call per traced event
is out of question because this would severely influence the flow of
control (as the system call may trigger the kernel to take a scheduling
decision) and the execution time of the traced code, not to speak of
the TLB and cache footprint.
* Kernel independence: We want to use the same tracing facility across
all supported base platforms.
* Accountability of resources: It must be clearly defined where the
resources for trace buffers come from. Ideally, the tracing tool should be
able to dimension the buffers according to its needs and, in turn, pay for
the buffers.
* Suitable level of abstraction: Only if the trace contains information at
the right level of abstraction, it can be interpreted for large scenarios.
A counter example is the in-kernel trace buffer of the Fiasco.OC kernel,
which logs kernel-internal object names and a few message words when tracing
IPC messages, but makes it almost impossible to map this low-level
information to the abstraction of the control flow of RPC calls. In
contrast, we'd like to capture the names of invoked RPC calls (which is an
abstraction level the kernel is not aware of). This requirement implies the
need to have useful trace points generated automatically. Ideally those
trace points should cover all interactions of a component with the outside
world.
* (Re-)definition of tracing policies at runtime: The
question of which information to gather when a trace point is passed
should not be solely defined at compile time. Instead of changing static
instrumentations in the code, we'd prefer to have a way to configure
the level of detail and possible conditions for capturing events at runtime,
similar to dtrace. This way, a series of different hypotheses could be
tested by just changing the tracing policy instead of re-building and
rebooting the entire scenario.
* Straight-forward implementation: We found that most existing tracing
solutions are complicated. For example, dtrace comes with a virtual
machine for the sandboxed interpretation of policy code. Another typical
source of complexity is the synchronization of trace-buffer accesses.
Because for Genode, low TCB complexity is of utmost importance, the
simplicity of the implementation is the prerequisite to make it an
integral part of the base system.
* Support for both online and offline analysis of traces.
We are happy to report to have come up with a design that meets all those
requirements thanks to the architecture of Genode. In the following, we
present the key aspects of the design.
The tracing facility comes in the form of a new TRACE service implemented
in core. Using this service, a TRACE client can gather information about
available tracing subjects (existing or no-longer existing threads),
define trace buffers and policies and assign those to tracing subjects,
obtain access to trace-buffer contents, and control the tracing state
of tracing subjects. When a new thread is created via a CPU session, the
thread gets registered at a global registry of potential tracing sources. Each
TRACE service manages a session-local registry of so-called trace subjects.
When requested by the TRACE client, it queries new tracing sources from the
source registry and obtains references to the corresponding threads. This way,
the TRACE session becomes able to control the thread's tracing state.
To keep the tracing overhead as low as possible, we assign a separate trace
buffer to each individually traced thread. The trace buffer is a shared memory
block mapped to the virtual address space of the thread's process. Capturing
an event comes down to a write operation into the thread-local buffer. Because
of the use of shared memory for the trace buffer, no system call is needed and
because the buffer is local to the traced thread, there is no need for
synchronizing the access to the buffer. When no tracing is active, a thread
has no trace buffer. The buffer gets installed only when tracing is started.
The buffer is not installed magically from the outside of the traced process
but from the traced thread itself when passing a trace point. To detect
whether to install a new trace buffer, there exists a so-called trace-control
dataspace shared between the traced process and its CPU service. This
dataspace contains control bits for each thread created via the CPU session.
The control bits are evaluated each time a trace point is passed by the
thread. When the thread detects a change of the tracing state, it actively
requests the new trace buffer from the CPU session and installs it into its
address space. The same technique is used for loading the code for tracing
policies into the traced process. The traced thread actively checks for
policy-version updates by evaluating the trace-control bits. If an update is
detected, the new policy code is requested from the CPU session. The policy
code comes in the form of position-independent code, which gets mapped into
the traced thread's address space by the traced thread itself. Once mapped,
a trace point will call the policy code. When called, the policy
module code returns the data to be captured into the trace buffer. The
relationship between the trace monitor (the client of TRACE service), core's
TRACE service, core's CPU service, and the traced process is depicted in
Figure [trace_control].
[image trace_control]
There is one trace-control dataspace per CPU session, which gets accounted
to the CPU session's quota. The resources needed for the
trace-buffer dataspaces and the policy dataspaces are paid-for by the
TRACE client. On session creation, the TRACE client can specify the amount
of its own RAM quota to be donated to the TRACE service in core. This
enables the TRACE client to define trace buffers and policies of arbitrary
sizes, limited only by its own RAM quota.
In Genode, the interaction of a process with its outside world is
characterized by its use of inter-process communication, namely synchronous
RPC, signals, and access to shared memory. For the former two types of
inter-process communication, Genode generates trace points automatically. RPC
clients generate trace points when an RPC call is issued and when a call
returned. RPC servers generate trace points in the RPC dispatcher, capturing
incoming RPC requests as well as RPC replies. Thanks to Genode's RPC
framework, we are able to capture the names of the RPC functions in the RPC
trace points. This information is obtained from the declarations of the RPC
interfaces. For signals, trace points are generated for submitting and
receiving signals. Those trace points form a useful base line for gathering
tracing data. In addition, manual trace points can be inserted into the code.
State of the implementation
===========================
The implementation of Genode's tracing facility is surprisingly low complex.
The addition to the base system (core as well as the base library) are
merely 1500 lines of code. The mechanism works across all base platforms.
Because the TRACE client provides the policy code and trace buffer to the
traced thread, the TRACE client imposes ultimate control over the traced
thread. In contrast to dtrace, which sandboxes the trace policy, we express
the policy module in the form of code executed in the context of the traced
thread. However, in contrast to dtrace, such code is never loaded into a large
monolithic kernel, but solely into the individually traced processes. So the
risk of a misbehaving policy is constrained to the traced process.
In the current form, the TRACE service of core should be considered as a
privileged service because the trace-subject namespace of each session
contains all threads of the system. Therefore, TRACE sessions should be routed
only for trusted processes. In the future, we plan to constrain the
namespaces for tracing subjects per TRACE session.
The TRACE session interface is located at base/include/trace_session/.
A simple example for using the service is available at os/src/test/trace/
and is accompanied with the run script os/run/trace.run. The test
demonstrates the TRACE session interface by gathering a trace of a thread
running locally in its address space.
Enhanced multi-processor support
################################
Multi-processor (MP) support is one of those features that most users take for
granted. MP systems are so ubiquitous, even on mobile platforms, that a
limitation to utilizing a single CPU only is almost a fallacy.
That said, MP support in operating systems is hard to get right. For this
reason, we successively deferred the topic on the agenda of Genode's
road map.
For some base platforms such as the Linux or Codezero kernels, Genode
always used to support SMP because the kernel would manage the affinity
of threads to CPU cores transparently to the user-level process. So on
these kernels, there was no need to add special support into the framework.
However, on most microkernels, the situation is vastly different. The
developers of such kernels try hard to avoid complexity in the kernel and
rightfully argue that in-kernel affinity management would contribute to kernel
complexity. Another argument is that, in contrast to monolithic kernels that
have a global view on the system and an "understanding" of the concerns of the
user processes, a microkernel is pretty clueless when it comes to the roles
and behaviours of individual user-level threads. Not knowing whether a thread
works as a device driver, an interactive program, or a batch process, the
microkernel is not in the position to form a reasonably useful model of the
world, onto which it could intelligently apply scheduling and affinity
heuristics. In fact, from the perspective of a microkernel, each thread does
nothing else than sending and receiving messages, and causing page faults.
For these reasons, microkernel developers tend to provide the bootstrapping
procedure for the physical CPUs and a basic mechanism to assign
threads to CPUs but push the rest of the problem to the user space, i.e.,
Genode. The most straight-forward way would make all physical CPUs visible
to all processes and require the user or system integrator to assign
physical CPUs when a thread is created. However, on the recursively
structured Genode system, we virtualize resources at each level, which
calls for a different approach. Section [Management of CPU affinities]
explains our solution.
When it comes to inter-process communication on MP systems, there is a
certain diversity among the kernel designs. Some kernels allow the user land
to use synchronous IPC between arbitrary threads, regardless of whether both
communication partners reside on the same CPU or on two different CPUs. This
convenient model is provided by Fiasco.OC. However, other kernels do not offer
any synchronous IPC mechanism across CPU cores at all, NOVA being a poster
child of this school of thought. If a user land is specifically designed for
a particular kernel, those peculiarities can be just delegated to the
application developers. For example, the NOVA user land called NUL is designed
such that a recipient of IPC messages spawns a dedicated thread on each
physical CPU. In contrast, Genode is meant to provide a unified API that
works well across various different kernels. To go forward, we had four
options:
# Not fully supporting the entity of API semantics across all base platforms.
For example, we could stick with the RPC API for synchronous communication
between threads. Programs just would happen to fail on some base platforms
when the called server resides on a different CPU. This would effectively
push the problem to the system integrator. The downside would be the
sacrifice of Genode's nice feature that a program developed
on one kernel usually works well on other kernels without any changes.
# Impose the semantics provided by the most restrictive kernel onto all
users of the Genode API. Whereas this approach would facilitate that
programs behave consistently across all base platforms, the restrictions
would be artificially imposed onto all Genode users, in particular the
users of kernels with less restrictions. Of course, we don't change
the Genode API lightheartedly, which attributes to our hesitance to go
into this direction.
# Hiding kernel restrictions behind the Genode API. This approach could come
in many different shapes. For example, Genode could transparently spawn a
thread on each CPU when a single RPC entrypoint gets created, following
the model of NUL. Or Genode could emulate synchronous IPC using the core
process as a proxy.
# Adapting the kernel to the requirements of Genode. That is, persuading
kernel developers to implement the features we find convenient, i.e., adding
a cross-CPU IPC feature to NOVA. History shows that our track record in
doing that is not stellar.
Because each of those options is like opening a different can of worms, we
used to defer the solution of the problem. Fortunately, however, we finally
kicked off a series of practical experiments, which led to a fairly elegant
solution, which is detailed in Section
[Adding multi-processor support to Genode on NOVA].
Management of CPU affinities
============================
In line with our experience of supporting
[https://genode.org/documentation/release-notes/10.02#Real-time_priorities - real-time priorities]
in version 10.02, we were seeking a way to express CPU affinities such that
Genode's recursive nature gets preserved and facilitated. Dealing with
physical CPU numbers would contradict with this mission. Our solution
is based on the observation that most MP systems have topologies that can
be represented on a two-dimensional coordinate system. CPU nodes close
to each other are expected to have closer relationship than distant nodes.
In a large-scale MP system, it is natural to assign clusters of closely
related nodes to a given workload. Genode's architecture is based on the
idea to recursively virtualize resources and thereby lends itself to the
idea to apply this successive virtualization to the problem of clustering
CPU nodes.
In our solution, each process has a process-local view on a so-called affinity
space, which is a two-dimensional coordinate space. If the process creates a
new subsystem, it can assign a portion of its own affinity space to the new
subsystem by imposing a rectangular affinity location to the subsystem's CPU
session. Figure [affinity_space] illustrates the idea.
[image affinity_space]
Following from the expression of affinities as a rectangular location within a
process-local affinity space, the assignment of subsystems to CPU nodes
consists of two parts, the definition of the affinity space dimensions as used
for the process and the association of sub systems with affinity locations
(relative to the affinity space). For the init process, the affinity space is
configured as a sub node of the config node. For example, the following
declaration describes an affinity space of 4x2:
! <config>
! ...
! <affinity-space width="4" height="2" />
! ...
! </config>
Subsystems can be constrained to parts of the affinity space using the
'<affinity>' sub node of a '<start>' entry:
! <config>
! ...
! <start name="loader">
! <affinity xpos="0" ypos="1" width="2" height="1" />
! ...
! </start>
! ...
! </config>
As illustrated by this example, the numbers used in the declarations for this
instance of the init process are not directly related to physical CPUs. If
the machine has just two cores, init's affinity space would be mapped
to the range [0,1] of physical CPUs. However, in a machine
with 16x16 CPUs, the loader would obtain 8x8 CPUs with the upper-left
CPU at position (4,0). Once a CPU session got created, the CPU client can
request the physical affinity space that was assigned to the CPU session
via the 'Cpu_session::affinity()' function. Threads of this CPU session
can be assigned to those physical CPUs via the 'Cpu_session::affinity()'
function, specifying a location relative to the CPU-session's affinity space.
Adding multi-processor support to Genode on NOVA
================================================
The NOVA kernel has been supporting MP systems for a long time. However
Genode did not leverage this capability until now. The main reason was that
the kernel does not provide - intentionally by the kernel developer - the
possibility to perform synchronous IPC between threads residing on different
CPUs.
To cope with this situation, Genode servers and clients would need to make
sure to have at least one thread on a common CPU in order to communicate.
Additionally, shared memory and semaphores could be used to communicate across
CPU cores. Both options would require rather fundamental changes to the Genode
base framework and the API. An exploration of this direction should in any
case be pursued in evolutionary steps rather than as one big change, also
taking into account that other kernels do not impose such hard requirements on
inter-CPU communication. To tackle the challenge, we conducted a series of
experiments to add some kind of cross-CPU IPC support to Genode/NOVA.
As a general implication of the missing inter-CPU IPC, messages between
communication partners that use disjoint CPUs must take an indirection through
a proxy process that has threads running on both CPUs involved. The sender
would send the message to a proxy thread on its local CPU, the proxy process
would transfer the message locally to the CPU of the receiver by using
process-local communication, and the proxy thread on the receiving CPU would
deliver the message to the actual destination. We came up with three options
to implement this idea prototypically:
# Core plays the role of the proxy because it naturally has access to all
CPUs and emulates cross-CPU IPC using the thread abstractions of the
Genode API.
# Core plays the role of the proxy but uses NOVA system calls directly
rather than Genode's thread abstraction.
# The NOVA kernel acts as the proxy and emulates cross-CPU IPC directly
in the kernel.
After having implemented the first prototypes, we reached the following
conclusions.
For options 1 and 2 where core provides this service: If a client can not
issue a local CPU IPC, it asks core - actually the pager of the client
thread - to perform the IPC request. Core then spawns or reuses a proxy thread
on the target CPU and performs the actual IPC on behalf of the client. Option
1 and 2 only differ in respect to code size and the question to whom to
account the required resources - since a proxy thread needs a stack and some
capability selectors.
As one big issue for option 1 and 2, we found that in order to delegate
capabilities during the cross-CPU IPC, core has to receive capability mappings
to delegate them to the target thread. However, core has no means to know
whether the capabilities must be maintained in core or not. If a capability is
already present in the target process, the kernel would just translate the
capability to the target's capability name space. So core wouldn't need to
keep it. In the other case where the target receives a prior unknown
capability, the kernel creates a new mapping. Because the mapping gets
established by the proxy in core, core must not free the capability.
Otherwise, the mapping would disappear in the target process. This means that
the use of core as a proxy ultimately leads to leaking kernel resources
because core needs to keep all transferred capabilities, just for the case a
new mapping got established.
For option 3, the same general functionality as for option 1 and 2 is
implemented in the kernel instead of core. If a local CPU IPC call
fails because of a BAD_CPU kernel error code, the cross-CPU IPC extension
will be used. The kernel extension creates - similar as to option 1 and 2 - a
semaphore (SM), a thread (EC), and a scheduling context (SC) on the remote
CPU and lets it run on behalf of the caller thread. The caller thread
gets suspended by blocking on the created semaphore until the remote EC has
finished the IPC. The remote proxy EC reuses the UTCB of the suspended caller
thread as is and issues the IPC call. When the proxy EC returns, it wakes up
the caller via the semaphore. Finally, the proxy EC and SC de-schedule
themselves and the resources get to be destroyed later on by the kernel's RCU
mechanism. Finally, when the caller thread got woken up, it takes care to
initiate the deconstruction of the semaphore.
The main advantage of option 3 compared to options 1 and 2 is that we don't
have to keep and track the capability delegations during a cross-CPU IPC.
Furthermore, we do not have potentially up to two additional address space
switches per cross-CPU IPC (from client to core and core to the server).
Additionally, the UTCB of the caller is reused by the proxy EC and does not
need to be allocated separately as for option 1 and 2.
For these reasons, we decided to go for the third option. From Genode's API
point of view, the use of cross-CPU IPC is completely transparent. Combined
with the affinity management described in the previous section, Genode/NOVA
just works on MP systems.
As a simple example for using Genode on MP systems, there is a ready-to-use
run script available at base/run/affinity.run.
Base framework
##############
Affinity propagation in parent, root, and RPC entrypoint interfaces
===================================================================
To support the propagation of CPU affinities with session requests, the
parent and root interfaces had to be changed. The 'Parent::Session'
and 'Root::Session' take the affinity of the session as a new argument.
The affinity argument contains both the dimensions of the affinity space
used by the session and the session's designated affinity location within
the space. The corresponding type definitions can be found at
base/affinity.h.
Normally, the 'Parent::Session' function is not used directly but indirectly
through the construction of a so-called connection object, which represents an
open session. For each session type there is a corresponding connection type,
which takes care of assembling the session-argument string by using the
'Connection::session()' convenience function. To maintain API compatibility,
we kept the signature of the existing 'Connection::session()' function using a
default affinity and added a new overload that takes the affinity as
additional argument. Currently, this overload is used in
cpu_session/connection.h.
For expressing the affinities of RPC entrypoints to CPUs within the affinity
space of the server process, the 'Rpc_entrypoint' takes the desired affinity
location of the entrypoint as additional argument. For upholding API
compatibility, the affinity argument is optional.
CPU session interface
=====================
The CPU session interface underwent changes to accommodate the new event
tracing infrastructure and the CPU affinity management.
Originally the 'Cpu_session::num_cpus()' function could be used to
determine the number of CPUs available to the session. This function
has been replaced by the new 'affinity_space' function, which returns the
bounds of the CPU session's physical affinity space. In the simplest case of
an SMP machine, the affinity space is one-dimensional where the width
corresponds to the number of CPUs. The 'affinity' function, which is used to
bind a thread to a specified CPU, has been changed to take an affinity
location as argument. This way, the caller can principally express the
affiliation of the thread with multiple CPUs to guide load-balancing in a CPU
service.
New TRACE session interface
===========================
The new event tracing mechanism as described in Section
[Light-weight event tracing] is exposed to Genode processes in the form
of the TRACE service provided by core. The new session interface
is located under base/include/trace_session/. In addition to the new session
interface, the CPU session interface has been extended with functions for
obtaining the trace-control dataspace for the session as well as the trace
buffer and trace policy for a given thread.
Low-level OS infrastructure
###########################
Event-driven operation of NIC bridge
====================================
The NIC bridge component multiplexes one physical network device among
multiple clients. It enables us to multiplex networking on the network-packet
level rather than the socket level and thereby take TCP/IP out of the
critical software stack for isolating network applications. As it
represents an indirection in the flow of all networking packets, its
performance is important.
The original version of NIC bridge was heavily multi-threaded. In addition to
the main thread, a timer thread, and a thread for interacting with the NIC
driver, it employed one dedicated thread per client. By merging those flows of
control into a single thread, we were able to significantly reduce the number
of context switches and improve data locality. These changes reduced the
impact of the NIC bridge on the packet throughput from 25% to 10%.
Improved POSIX thread support
=============================
To accommodate qtwebkit, we had to extend Genode's pthread library with
working implementations of condition variables, mutexes, and thread-local
storage. The implemented functions are attr_init, attr_destroy, attr_getstack,
attr_get_np, equal, mutex_attr, mutexattr_init, mutexattr_destroy,
mutexattr_settype, mutex_init, mutex_destroy, mutex_lock, mutex_unlock,
cond_init, cond_timedwait, cond_wait, cond_signal, cond_broadcast, key_create,
setspecific, and getspecific.
Device drivers
##############
SATA 3.0 on Exynos 5
====================
The previous release featured the initial version of our SATA 3.0 driver for
the Exynos 5 platform. This driver located at os/src/drivers/ahci/exynos5 has
reached a fully functional state by now. It supports UDMA-133 with up to
6 GBit/s.
For driver development, we set the goal to reach a performance equal to
the Linux kernel. To achieve that goal, we had to make sure to
operate the controller and the disks in the same ways as Linux does.
For this reason, we modeled our driver closely after the behaviour of the
Linux driver. That is, we gathered traces of I/O transactions to determine the
initialization steps and the request patterns that Linux performs to access
the device, and used those behavioral traces as a guide for our
implementation. Through step-by-step analysis of the traces, we not only
succeeded to operate the device in the proper modes, but we also found
opportunities for further optimization, in particular regarding the error
recovery implementation.
This approach turned out to be successful. We measured that our driver
generally operates as fast (and in some cases even significantly faster)
than the Linux driver on solid-state disks as well as on hard disks.
Dynamic CPU frequency scaling for Exynos 5
==========================================
As the Samsung Exynos-5 SoC is primarily targeted at mobile platforms,
power management is an inherent concern. Until now, Genode did not pay
much attention to power management though. For example, we completely
left out the topic from the scope of the OMAP4 support. With the current
release, we took the first steps towards proper power management on ARM-based
platforms in general, and the Exynos-5-based Arndale platform in particular.
First, we introduced a general interface to regulate clocks and voltages.
Priorly, each driver did its own part of configuring clock and power control
registers. The more device drivers were developed, the higher were the chances
that they interfere when accessing those clock, or power units.
The newly introduced "Regulator" interface provides the possibility to enable
or disable, and to set or get the level of a regulator. A regulator might be
a clock for a specific device (such as a CPU) or a voltage regulator.
For the Arndale board, an exemplary implementation of the regulator interface
exists in the form of the platform driver. It can be found at
os/src/drivers/platform/arndale. Currently, the driver implements
clock regulators for the CPU, the USB 2.0 and USB 3.0 host controller, the
eMMC controller, and the SATA controller. Moreover, it provides power
regulators for SATA, USB 2.0, and USB 3.0 host controllers. The selection of
regulators is dependent on the availability of drivers for the platform.
Otherwise it wouldn't be possible to test that clock and power state doesn't
affect the device.
Apart from providing regulators needed by certain device drivers, we
implemented a clock regulator for the CPU that allows changing the CPU
frequency dynamically and thereby giving the opportunity to scale down
voltage and power consumption. The possible values range from 200 MHz to
1.7 GHz whereby the last value isn't recommended and might provoke system
crashes due to overheating. When using Genode's platform driver for Arndale
it sets CPU clock speed to 1.6 GHz by default. When reducing
the clock speed to the lowest level, we observed a power consumption
reduction of approximately 3 Watt. Besides reducing dynamic power consumption
by regulating the CPU clock frequency, we also explored the gating of the clock
management and power management to further reduce power consumption.
With the CPU frequency scaling in place, we started to close all clock gates
not currently in use. When the platform driver for the Arndale board gets
initialized, it closes everything. If a device driver enables its clock
regulator, all necessary clock gates for the device's clock are opened. This
action saves about 0.7 Watt. The initial closing of all unnecessary power
gates was much more effective. Again, everything not essential for the working
of the kernel is disabled on startup. When a driver enables its power
regulator, all necessary power gates for the device are opened. Closing all
power gates saves about 2.6 Watt.
If we consider all measures taken to save power, we were able to reduce power
consumption to about 59% without performance degradation. When measuring power
consumption after boot up, setting the CPU clock to 1.6 GHz, and fully load
both CPU cores without the described changes, we measured about 8 Watt. With
the described power saving provisions enabled, we measured about 4.7 Watt.
When further reducing the CPU clock frequency to 200 MHz, only 1.7 Watt were
measured.
VESA driver moved to libports
=============================
The VESA framebuffer driver executes the initialization code located
in VESA BIOS of the graphics card. As the BIOS code is for real mode,
the driver uses the x86emu library from X11 as emulation environment.
We updated x86emu to version 1.20 and moved the driver from the 'os'
repository to the 'libports' repository as the library is third-party
code. Therefore, if you want to use the driver, the 'libports'
repository has to be prepared
('make -C <genode-dir>/libports prepare PKG=x86emu') and enabled in
your 'etc/build.conf'.
Runtime environments
####################
Seoul (aka Vancouver) VMM on NOVA
=================================
Since we repeatedly received requests for using the Seoul respectively
Vancouver VMM on NOVA, we improved the support for this virtualization
solution on Genode. Seoul now supports booting from raw hard disk images
provided via Genode's block session interface. Whether this image is actually
a file located in memory, or it is coming directly from the hard disk, or just
from a partition of the hard disk using Genode's part_blk service, is
completely transparent thanks to Genode's architecture.
Additionally, we split up the one large Vancouver run script into several
smaller Seoul run scripts for easier usage - e.g. one for disk, one for
network testing, one for automated testing, and one we call "fancy". The
latter resembles the former vancouver.run script using Genode's GUI to let the
user start VMs interactively. The run scripts prefixed with 'seoul-' can be
found at ports/run. For the fancy and network scripts, ready-to-use VM images
are provided. Those images are downloaded automatically when executing the run
script for the first time.
L4Linux on Fiasco.OC
====================
L4Linux has been updated from version 3.5.0 to Linux kernel version 3.9.0 thus
providing support for contemporary user lands running on top of L4Linux on both
x86 (32bit) and ARM platforms.
Noux runtime for Unix software
==============================
Noux is our way to use the GNU software stack natively on Genode. To improve
its performance, we revisited the address-space management of the runtime to
avoid redundant revocations of memory mappings when Noux processes are cleaned
up.
Furthermore, we complemented the support for the Genode tool chain to
cover GNU sed and GNU grep as well. Both packages are available at the ports
repository.
Platforms
#########
Fiasco.OC updated to revision r56
=================================
Fiasco.OC and the required L4RE parts have been updated to the current SVN
revision (r56). For us, the major new feature is the support of Exynos SOCs in
the mainline version of Fiasco.OC (www.tudos.org). Therefore Genode's
implementation of the Exynos5250 platform could be abandoned leading to less
maintenance overhead of Genode on Fiasco.OC.
Furthermore, Genode's multi-processor support for this kernel has been
improved so that Fiasco.OC users benefit from the additions described in
Section [Enhanced multi-processor support].
NOVA updated
============
In the process of our work on the multi-processor support on NOVA, we updated
the kernel to the current upstream version. Additionally, our customized branch
(called r3) comes with the added cross-CPU IPC system call and improvements
regarding the release of kernel resources.
Integrity checks for downloaded 3rd-party software
##################################################
Even though Genode supports a large variety of 3rd-party software, its
source-code repository contains hardly any 3rd-party source code. Whenever
3rd-party source code is needed, Genode provides automated tools for
downloading the code and integrating it with the Genode environment. As of
now, there exists support for circa 70 software packages, including the
tool chain, various kernels, libraries, drivers, and a few applications. Of
those packages, the code for 13 packages comes directly from their respective
Git repositories. The remaining 57 packages are downloaded in the form of tar
archives from public servers via HTTP or FTP. Whereas we are confident with
the integrity of the code that comes from Git repositories, we are less so
about the archives downloaded from HTTP or FTP servers.
Fortunately, most Open-Source projects provide signature files that allow
the user to verify the origin of the archive. For example, archives of
GNU software are signed with the private key of the GNU project. So the
integrity of the archive can be tested with the corresponding public key.
We used to ignore the signature files for many years but
this has changed now. If there is a signature file available for a package,
the package gets verified right after downloading. If only a hash-sum file
is provided, we check it against a known-good hash sum.
The solution required three steps, the creation of tools for validating
signatures and hashes, the integration of those tools into Genode's
infrastructure for downloading the 3rd-party code, and the definition of
verification rules for the individual packages.
First, new tools for downloading and validating hash sums and signatures were
added in the form of the shell scripts download_hashver (verify hash sum) and
download_sigver (verify signature) found at the tool/ directory. Under the
hood, download_sigver uses GNU GPG, and download_hashver uses the tools
md5sum, sha1sum, and sha256sum provided by coreutils.
Second, hooks for invoking the verification tools were added to the
tool-chain build script as well as the ports and the libports repositories.
The third and the most elaborative step, was going through all the packages,
looking for publicly available signature files, and adding corresponding
package rules. As of now, this manual process has been carried out for 30
packages, thereby covering the half of the archives.
Thanks to Stephan Mueller for pushing us into the right direction, kicking off
the work on this valuable feature, and for the manual labour of revisiting all
the 3rd-party packages!

0
doc/release_notes-13-11.txt → doc/release_notes/13-11.txt
View File

791
doc/release_notes/14-02.txt Normal file
View File

@@ -0,0 +1,791 @@
===============================================
Release notes for the Genode OS Framework 14.02
===============================================
Genode Labs
During the release cycle of version 14.02, our development has been focused on
storage and virtualization. It goes without saying that proper support for
block-device access and file systems is fundamental for the use of
Genode as general-purpose OS. Virtualization is relevant as well because
it bridges the gap between the functionality we need and the features
natively available on Genode today.
Our work on the storage topic involved changes of the block-driver APIs to an
asynchronous mode of operation, overhauling most of the existing block-level
components, as well as the creation of new block services, most importantly a
block cache. At file-system level, we continued our line of work on FUSE-based
file systems, adding support for NTFS-3g. A new highlight, however, is a new
file-system service that makes the file systems of the NetBSD kernel available
to Genode. This is made possible by using rump kernels as described in Section
[NetBSD file systems using rump kernels].
Virtualization on Genode has a long history, starting with the original
support of OKLinux on the OKL4 kernel (OKLinux is no longer supported), over
the support of L4Linux on top of the Fiasco.OC kernel, to the support of the
Vancouver VMM on top of NOVA. However, whereas each of those variants has
different technical merits, all of them were developed in the context of
university research projects and were never exposed to real-world scenarios.
We were longing for a solution that meets the general expectations from a
virtualization product, namely the support for a wide range of guest OSes,
guest-host integration features, ease of use, and an active development.
VirtualBox is one of the most popular commodity virtualization products as of
today. With the current release, we are happy to announce the availability of
VirtualBox on top of Genode/NOVA. Section
[VirtualBox on top of the NOVA microhypervisor] gives insights into the
background of this development, the technical challenges we had to overcome,
and the current state of the implementation.
In addition to addressing storage and virtualization, the current release
comes with a new pseudo file system called trace_fs that allows the
interactive use of Genode's tracing facilities via Unix commands,
a profound unification of the various graphics back ends used throughout
the framework, a new facility for propagating status reports, and
improvements of the Noux runtime for executing Unix software on Genode.
VirtualBox on top of the NOVA microhypervisor
#############################################
Virtualization is an important topic for Genode for two distinct reasons.
It is repeatedly requested by users of the framework who consider
Genode as a microkernel-based hosting platform for virtual machines,
and it provides a smooth migration path from using Linux-based systems
towards using Genode as day-to-day OS.
Why do people consider Genode as a hosting platform for virtual machines
if there is an abundance of mature virtualization solutions on the market?
What all existing popular solutions have in common is the staggering complexity
of their respective trusted-computing base (TCB). The user of a virtual
machine on a commodity hosting platform has to trust millions of lines of
code. For example, with Xen, the TCB comprises the hypervisor and the Linux
system running as DOM0. For security-sensitive application areas, it is
almost painful to trust such a complex foundation. In contrast, the TCB of a
hosting platform based on Genode/NOVA is two orders of magnitude less complex.
Lowering the complexity reduces the likelihood for vulnerabilities and thereby
mitigates the attack surface of the system. It also enables the assessment of
security properties by thorough evaluation or even formal verification. In the
light of the large-scale privacy issues of today, the desire for systems that
are resilient against malware and zero-day exploits has never been higher.
Microkernel-based operating systems promise a solution. Virtualization enables
compatibility to existing software. Combining both seems natural. This is what
Genode/NOVA stands for.
From the perspective of us Genode developers who are in the process of
migrating from Linux-based OSes to Genode as day-to-day OS, we consider
virtualization as a stop-gap solution for all those applications that
do not exist natively on Genode, yet. Virtualization makes our transition
an evolutionary process.
Until now, NOVA was typically accompanied with a co-developed virtual machine
monitor called Seoul (formerly called Vancouver), which is executed as a
regular user-level process on top of NOVA. In contrast to conventional wisdom
about the performance of microkernel-based systems, the Seoul VMM on top of
NOVA is extremely fast, actually faster then most (if not all) commonly used
virtualization solutions. However, originating from a research project, Seoul
is quite challenging to use and not as mature as commodity VMMs that were
developed as real-world products. For example, there is a good chance that an
attempt to boot an arbitrary version of a modern Linux distribution might just
fail. In our experience, it takes a few days to investigate the issues, modify
the guest OS configuration, and tweak the VMM here and there, to run the OS
inside the Seoul VMM. That is certainly not a show stopper in appliance-like
scenarios, but it rules out Seoul as a general solution. Running Windows
OS as guest is not supported at all, which further reduces the application
areas of Seoul. With this in mind, it is unrealistic to propose the use
of Genode/NOVA as an alternative for popular VM hosting solutions.
Out of this realization, the idea was born to combine NOVA's virtualization
interface with a time-tested and fully-featured commodity VMM. Out of the
available Open-Source virtualization solutions, we decided to take a closer
look at VirtualBox, which attracted us for several reasons: First, it is
portable, supporting various host OSes such as Solaris, Windows OS, Linux,
and Mac OS X. Second, it has all the guest-integration features we could
wish for. There are extensive so-called guest additions for popular guest
OSes that vastly improve the guest-OS performance and allow a tight
integration with the host OS using shared folders or a shared clipboard.
Third, it comes with sophisticated device models that support all
important popular guest OSes. And finally, it is actively developed and
commercially supported.
However, moving VirtualBox over to NOVA presented us with a number of
problems. As a precondition, we needed to gain a profound understanding
of the VirtualBox architecture and the code base. To illustrate the challenge,
the source-code distribution of VirtualBox comprises 2.8 million lines of
code. This code contains build tools, the VMM, management tools, several
3rd-party libraries, middleware, the guest additions, and tests. The pieces
that are relevant for the actual VMM amount to 700 thousand lines. By
reviewing the architecture, we found that the part of VirtualBox that
implements the hypervisor functionality (the world switch) runs in the
kernel of the host OS (it is loaded on demand by the user-level VM process
through the _/dev/vboxdrv_ interface into the host OS kernel). It is
appropriately named VMMR0. Once installed into the host OS kernel, it
takes over the control over the machine. To put it blatantly simple, it runs
"underneath" the host OS. The VMMR0 code is kernel agnostic, which explains
the good portability of VirtualBox across various host OSes. Porting
VirtualBox to a new host OS comes down to finding a hook for installing the
VMMR0 code into the host OS kernel and adapting the VirtualBox runtime API
to the new host OS.
In the context of microkernel-based systems, however, it becomes clear that
this classical approach of porting VirtualBox would subvert the microkernel
architecture. Not only would we need to punch a hole into NOVA for loading
additional kernel code, but also the VMMR0 code would inflate the amount of
code executed in privileged mode by more than factor 20. Both implications
are gross violations of the microkernel principle. Consequently, we needed to
find a different way to marry NOVA with VirtualBox.
Our solution was the creation of a drop-in replacement of the VMMR0 code that
runs solely at user level and interacts with NOVA's virtualization
interface. Our VMMR0 emulation code is co-located with the VirtualBox
VM process. Architecturally, the resulting solution is identical to the
use of Seoul on top of NOVA. There is one VM process per virtual machine,
and each VM process is isolated from others by the NOVA kernel. In
addition to creating the VMMR0 emulation code, we needed to replace some parts
of the VirtualBox VMMR3 code with custom implementations because they
overlapped with functionality provided by NOVA's virtualization interface,
in particular the provisioning of guest-physical memory. Finally, we needed
to interface the VM process with Genode's API to let the VM process
interact with Genode's input, file-system, and framebuffer services.
The result of this undertaking is available at the _ports_ repository.
VirtualBox can be downloaded and integrated with Genode via the following
command issued from within the repository:
! make prepare PKG=virtualbox
To illustrate the integration of VirtualBox into a Genode system, there
is run script located at _ports/run/virtualbox.run_. It expects a
bootable ISO image containing a guest OS at _<build-dir>/bin/test.iso_.
The configuration of the VirtualBox process is as simple as
! <config>
! <image type="iso" file="/iso/test.iso" />
! </config>
VirtualBox will try to obtain the specified ISO file via a file-system
session. Furthermore, it will open a framebuffer session and an input session.
The memory assigned to the guest OS depends on the RAM quota assigned to the
VirtualBox process. Booting a guest OS stored in a VDI file is supported. The
image type must be changed to "vdi" accordingly.
Please note that this first version of VirtualBox is far from being complete
as it lacks many features (SMP, guest-addition support, networking), is not
optimized, and must be considered as experimental. However, we could
successfully run GNU/Linux, Android, Windows XP, Windows 7, HelenOS, Minix-3,
GNU Hurd, and of course Genode inside VirtualBox.
One point we are pretty excited about is that the porting effort to
Genode/NOVA did not require any change of Genode. From Genode's point of
view, VirtualBox is just an ordinary leaf node of the process tree, which
can happily co-exist with other processes - even if it is the Seoul VMM.
[image seoul-vbox-win7-tinycore]
In the screenshot above, VirtualBox is running besides the Seoul VMM on top of
Genode/NOVA. Seoul executes Tinycore Linux as guest OS. VirtualBox executes MS
Windows 7. Both VMMs are using hardware virtualization (VT-x) but are plain
user-level programs with no special privileges.
NetBSD file systems using rump kernels
######################################
In the previous release, we made FUSE-based file systems available to Genode
via a custom implementation of the FUSE API. Even though this step made
several popular file systems available, we found that the file systems most
important to us (such as ext) are actually not well supported by FUSE. For
example, write support on ext2 is declared as an experimental feature. In
hindsight it is clear why: FUSE is primarily being used for accessing file
systems not found in the Linux kernel. So it shines with supporting NTFS
but less so with file systems that are well supported by the Linux kernel.
Coincidentally, when we came to this realization, we stumbled upon the
wonderful work of Antti Kantee on so-called rump kernels:
:[https://wiki.netbsd.org/rumpkernel/]:
Rump kernel Wiki
The motivation behind the rump kernels was the development of
NetBSD kernel subsystems (referred to as "drivers") in the NetBSD user land.
Such subsystems like file systems, device drivers, or the TCP/IP stack are
linked against a stripped-down version of the NetBSD kernel that can be
executed in user mode and uses a fairly small "hypercall" interface to
interact with the outside world. A rump kernel contains everything needed to
execute NetBSD kernel subsystems but hardly anything else. In particular, it
does not support the execution of programs on top. From our perspective,
having crafted device-driver environments (DDEs) for Linux, iPXE, and OSS over
the years, a rump kernel sounded pretty much like a DDE for NetBSD. So we
started exploring rump kernels with the immediate goal of making time-tested
NetBSD file systems available to Genode.
To our delight, the integration of rump kernels into the Genode system went
fairly smooth. The most difficult part was the integration of the NetBSD build
infrastructure with Genode's build system. The glue between rump kernels and
Genode is less than 3,000 lines of code. This code enables us to reuse all
NetBSD file systems on Genode. A rump kernel instance that contains several
file systems such as ext2, iso9660, msdos, and ffs takes about 8 MiB of memory
when executed on Genode.
The support for rump kernels comes in the form of the dedicated _dde_rump_
repository. For downloading and integrating the required NetBSD source code,
the repository contains a Makefile providing the usual 'make prepare'
mechanism. To build the file-system server, make sure to add the _dde_rump_
repository to the 'REPOSITORIES' declaration of your _etc/build.conf_ file
within your build directory. The server then can be built via
! make server/rump_fs
There is a run script located at _dde_rump/run/rump_ext2.run_ to execute
a simple test scenario:
! make run/rump_ext2
The server can be configured as follows:
!<start name="rump_fs">
! <resource name="RAM" quantum="8M" />
! <provides><service name="File_system"/></provides>
! <config fs="ext2fs"><policy label="" root="/" writeable="yes"/></config>
!</start>
On startup, it requests a service that provides a block session. If
there is more than one block session in the system, the block session must be
routed to the right block-session server. The value of the _fs_ attribute of
the '<config>' node can be one of the following: _ext2fs_ for EXT2, _cd9660_ for
ISO-9660, or _msdos_ for FAT file-system support. _root_ defines the directory
of the file system as seen as root directory by the client. The server hands
most of its RAM quota to the rump kernel. This means the larger the quota is,
the larger the internal block caches of the rump kernel will be.
Base framework
##############
The base API has not underwent major changes apart from the addition of
a few new utilities and minor refinements. Under the hood, however, the inner
workings of the framework received much attention, including an extensive
unification of the startup code and stack management.
New 'construct_at' utility
==========================
A new utility located at 'base/include/util/construct_at.h' allows for the
manual placement of objects without the need to have a global placement new
operation nor the need for type-specific new operators.
New utility for managing volatile objects
=========================================
Throughout Genode, we maintain a programming style that largely avoids dynamic
memory allocations. For the most part, higher-level objects aggregate
lower-level objects as class members. For example, the nitpicker GUI server
is actually a compound of such aggregations (see
[https://github.com/genodelabs/genode/blob/master/os/src/server/nitpicker/main.cc#L803 - Nitpicker::Main]).
This functional programming style leads to robust programs but it poses a
problem for programs that are expected to adopt their behaviour at runtime.
For the example of nitpicker, the graphics back end of the GUI server takes
the size of the screen as constructor argument. If the screen size changes,
the once constructed graphics back end becomes inconsistent with the new
screen size. We desire a way to selectively replace an aggregated object by a
new version with updated constructor arguments. The new utilities found in
'os/include/util/volatile_object.h' solve this problem. A so-called
'Volatile_object' wraps an object of the type specified as template argument.
In contrast of a regular object, a 'Volatile_object' can be re-constructed any
number of times by calling 'construct' with the constructor arguments. It is
accompanied with a so-called 'Lazy_volatile_object', which remains
unconstructed until 'construct' is called the first time.
Changed interface of 'Signal_rpc_member'
========================================
We unified the 'Signal_rpc_member' interface to be more consistent with the
'Signal_rpc_dispatcher'. The new version takes an entrypoint as argument and
cares for dissolving itself from the entrypoint when destructed.
Filename as default label for ROM connections
=============================================
Since the first version of Genode, ROM services used to rely on a "filename"
provided as session argument. In the meanwhile, we established the use of the
session label to select routing policies as well as server-side policies.
Strictly speaking, the name of a ROM module is used as a key to a server-side
policy of ROM services. So why not to use the session label to express the
key as we do with other services? By assigning the file name as label for ROM
sessions, we may become able to remove the filename argument in the future by
just interpreting the last part of the label as filename. By keeping only the
label, we won't need to consider conditional routing (via '<if-arg>') based on
session arguments other than the label anymore, which would simplify Genode
configurations in the long run. This change is transparent at API level but
may be taken into consideration when configuring Genode systems.
New 'Genode::Deallocator' interface
===================================
By splitting the new 'Genode::Deallocator' interface from the former
'Genode::Allocator' interface, we become able to restrict the accessible
operations for code that is only supposed to release memory, but not
perform any allocations.
Closely related to the allocator interface, we introduced variants of the
'new' operator that take a reference (as opposed to a pointer) to a
'Genode::Allocator' as argument.
Unified main-stack management and startup code among all platforms
==================================================================
In contrast to the stacks of regular threads, which are located within a
dedicated virtual-address region called thread-context area, the stack of
the main thread of a Genode program used to be located within the BSS
segment. If the stack of a normal thread overflows, the program produces
an unresolvable page fault, which can be easily debugged. However,
an overflowing main stack would silently corrupt the BSS segment. With
the current release, we finally resolved this long-standing problem by
moving the main stack to the context area, too. The tricky part was that
the context area is created by the main thread. So we hit a hen-and-egg
problem. We overcame this problem by splitting the process startup
into two stages, both called from the crt0 assembly code. The first
stage runs on a small stack within the BSS and has the sole purpose
of creating the context area and a thread object for the main thread.
This code path (and thereby the stack usage) is the same for all programs.
So we can safely dimension the stage-1 stack. Once the first stage
returns to the crt0 assembly code, the stack pointer is loaded with the
stack that is now located within the context area. Equipped with the
new stack, the actual startup code ('_main') including the global
constructors of the program is executed.
This change paved the ground for several further code unifications and
simplifications, in particular related to the dynamic linker.
Low-level OS infrastructure
###########################
Revised block-driver framework
==============================
Whereas Genode's block-session interface was designed to work asynchronously
and supports the out-of-order processing of requests, those capabilities
remained unused by the existing block services as those services used to
operate synchronously to keep their implementation simple. However, this
simplicity came at the prize of two disadvantages: First, it prevented us
to fully utilize native command queuing of modern disk controllers. Second,
when chaining components such as a block driver, the part_blk server, and
a file system, latencies accumulated along the chain of services. This
hurts the performance of random access patterns.
To overcome this limitation, we changed the block-component framework to work
asynchronously and to facilitate the recently introduced server API.
Consequently, all users of the API underwent an update. The affected
components are rom_loopdev, atapi_drv, fb_block_adapter, http_block, usb_drv,
and part_blk. For some components, in particular part_blk, this step led to a
complete redesign.
Besides the change of the block-component framework, the block-session
interface got extended to support logical block addresses greater than
32bit (LBA48). Thereby, the block component framework can now support
devices that exceed 2 TiB in size.
Block cache
===========
The provisioning of a block cache was one of the primary motivations behind the
[https://genode.org/documentation/release-notes/13.11#Dynamic_resource_balancing - dynamic resource balancing]
concept that was introduced in Genode 13.11. We are now introducing the first
version of such a cache.
The new block cache component located at _os/src/server/blk_cache/_ is both
a block-session client as well as a block-session server serving a single
client. It is meant to sit between a block-device driver and a file-system
server. When accessing the block device, it issues requests at a granularity
of 4K and thereby implicitly reads ahead whenever a client requests a smaller
amount of blocks. Blocks obtained from the device or written by the client
are kept in memory. If memory becomes scarce, the block cache first tries
to request further memory resources from its parent. If the request
gets denied, the cache evicts blocks from memory to the block device following
a least-recently-used replacement strategy. As of now, the block cache supports
dynamic resource requests to grow on demand but support for handling yield
requests is not yet implemented. So memory once handed out to the block cache
cannot be regained. Adding support for yielding memory on demand will be
complemented in the next version.
To see how to integrate the block cache in a Genode scenario, there is a
ready-to-use run script available at _os/run/blk_cache.run_.
File-system infrastructure
==========================
In addition to the integration of NetBSD's file systems, there are
file-system-related improvements all over the place.
First, the 'File_system::Session' interface has been extended with a 'sync'
RPC function. This function allows the client of a file system to force
the file system to write back its internal caches.
Second, we extended the FUSE implementation introduced with the previous
release.
Since file systems tend to have a built-in caching mechanism, we need to
sync these caches at the end of a session when using the fuse_fs server.
Therefore, each FUSE file system port has to implement a 'Fuse::sync_fs()'
function that executes the necessary actions if requested. Further
improvements are related to the handling of symbolic links and error
handling. Finally, we added a libc plugin for accessing NTFS file systems
via the ntfs-3g library.
Third, we complemented the family of FUSE-based libc plugins with a family of
FUSE-based file-system servers. To utilize a FUSE file system, there is a
dedicated binary (e.g., _os/src/server/fuse_fs/ext2_) for each FUSE
file-system server.
Note that write support is possible but considered to be experimental at this
point. For now, using it is not recommended.
To use the ext2_fuse_fs server in Noux, the following configuration snippet
may be used:
! <start name="ext2_fuse_fs">
! <resource name="RAM" quantum="8M"/>
! <provides> <service name="File_system"/> </provides>
! <config>
! <policy label="noux -> fuse" root="/" writeable="no" />
! </config>
! </start>
Finally, the libc file-system plugin has been extended to support 'unlink'.
Trace file system
=================
The new _trace_fs_ server provides access to a trace session by providing a
file-system session as front end. Combined with Noux, it allows for the
interactive exploration and tracing of Genode's process tree using
traditional Unix tools.
Each trace subject is represented by a directory ('thread_name.subject') that
contains specific files, which are used to control the tracing process of the
thread as well as storing the content of its trace buffer:
:'enable': The tracing of a thread is activated if there is a valid policy
installed and the intend to trace the subject was made clear by writing '1'
to the 'enable' file. The tracing of a thread may be deactivated by writing a
'0' to this file.
:'policy': A policy may be changed by overwriting the currently used one in the
'policy' file. In this case, the old policy is replaced by the new one and
automatically used by the framework.
:'buffer_size': Writing a value to the 'buffer_size' file changes the size of
the trace buffer. This value is evaluated only when reactivating the tracing
of the thread.
:'events': The trace-buffer contents may be accessed by reading from the
'events' file. New trace events are appended to this file.
:'active': Reading the file will return whether the tracing is active (1) or
not (0).
:'cleanup': Nodes of untraced subjects are kept as long as they do not change
their tracing state to dead. Dead untraced nodes are automatically removed
from the file system. Subjects that were traced before and are now untraced
can be removed by writing '1' to the 'cleanup' file.
To use the trace_fs, a configuration similar to the following may be used:
! <start name="trace_fs">
! <resource name="RAM" quantum="128M"/>
! <provides><service name="File_system"/></provides>
! <config>
! <policy label="noux -> trace"
! interval="1000"
! subject_limit="512"
! trace_quota="64M" />
! </config>
! </start>
:'interval': sets the period the Trace_session is polled. The
time is given in milliseconds.
:'subject_limit': specifies how many trace subjects should by acquired at
max when the Trace_session is polled.
:'trace_quota': is the amount of quota the trace_fs should use for the
Trace_session connection. The remaining amount of RAM quota will be used
for the actual nodes of the file system and the 'policy' as well as the
'events' files.
In addition, there are 'buffer_size' and 'buffer_size_limit' that define
the initial and the upper limit of the size of a trace buffer.
A ready-to-use run script can by found in 'ports/run/noux_trace_fs.run'.
Unified interfaces for graphics
===============================
Genode comes with several programs that perform software-based graphics
operations. A few noteworthy examples are the nitpicker GUI server,
the launchpad, the scout tutorial browser, or the terminal. Most of those
programs were equipped with their custom graphics back end. In some
cases such as the terminal, nitpicker's graphics back end was re-used.
But this back end is severely limited because its sole purpose is the
accommodation of the minimalistic (almost invisible) nitpicker GUI server.
The ongoing work on Genode's new user interface involves the creation of
new components that rely on a graphics back end. Instead of further
diversifying the zoo of graphics back ends, we took the intermediate step
to consolidate the existing back ends into one unified concept such that
application-specific graphics back ends can be created and extended using
modular building blocks. The new versions of nitpicker, scout, launchpad,
liquid_fb, nitlog, and terminal have been changed to use the new common
interfaces:
:os/include/util/geometry.h: Basic data structures and operations needed
for 2D graphics.
:os/include/util/color.h: Common color representation and utilities.
:os/include/os/pixel_rgba.h: Class template for representing a pixel.
:os/include/os/pixel_rgb565.h: Template specializations for RGB565 pixels.
:os/include/os/surface.h: Target surface, onto which graphics operations
can be applied.
:os/include/os/texture.h: Source texture for graphics operations that
transfer 2D pixel data to a surface.
The former _os/include/nitpicker_gfx/_ directory is almost deserted. The only
remainders are functors for the few graphics operations actually required by
nitpicker. For the scout widgets, the corresponding functors have become
available at the public headers at _demo/include/scout_gfx/_.
Because the scout widget set is used by at least three programs and will
most certainly play a role in new GUI components, we undertook a major
cleanup of the parts worth reusing. The result can be found at
_demo/include/scout/_.
New session interface for status reporting
==========================================
Genode has a uniform way of how configuration information is passed from
parents to children within the process tree by the means of "config" ROM
modules. Using this mechanism, a parent is able to steer the behaviour of
its children, not just at their start time but also during runtime.
Until now, however, there was no counterpart to the config mechanism, which
would allow a child to propagate runtime information to its parent. There
are many use cases for such a mechanism. For example, a bus-controller driver
might want to propagate a list of devices attached to the bus. When a new
device gets plugged in, this list should be updated to let the parent
take the new device resource into consideration. Another use case would be the
propagation of status information such as the feature set of a plugin.
Taken to the extreme, a process might expose its entire internal state to its
parent in order to allow the parent to kill and restart the process, and
feed the saved state back to the new process instance.
To cover these use cases, we introduced the new report-session interface. When
a client opens a report session, it transfers a part of its RAM quota to the
report server. In return, the report server hands out a dataspace dimensioned
according to the donated quota. Upon reception of the dataspace, the client
can write its status reports into the dataspace and inform the server about
the update via the 'submit' function. In addition to the mere reporting of
status information, the report-session interface is designed to allow the
server to respond to reports. For example, if the report mechanism is used to
implement a desktop notification facility, the user may interactively respond
to an incoming notification. This response can be reflected to the originator
of the notification via the 'response_sigh' and 'obtain_response' functions.
The new _report_rom_ component is both a report service and a ROM service. It
reflects incoming reports as ROM modules. The ROM modules are named
after the label of the corresponding report session.
Configuration
-------------
The report-ROM server hands out ROM modules only if explicitly permitted by a
configured policy. For example:
! <config>
! <rom>
! <policy label="decorator -> pointer" report="nitpicker -> pointer"/>
! <policy ... />
! ...
! </rom>
! </config>
The label of an incoming ROM session is matched against the 'label' attribute
of all '<policy>' nodes. If the session label matches a policy label, the
client obtains the data from the report client with the label specified in the
'report' attribute. In the example above, the nitpicker GUI server sends
reports about the pointer position to the report-ROM service. Those reports
are handed out to a window decorator (labeled "decorator") as ROM module.
XML generator utility
=====================
With the new report-session interface in place, comes the increased
need to produce XML data. The new XML generator utility located at
_os/include/util/xml_generator.h_ makes this extremely easy, thanks to
C++11 language features. For an example application, refer to
_os/src/test/xml_generator/_ and the corresponding run script at
_os/run/xml_generator.run_.
Dynamic ROM service for automated testing
=========================================
The new _dynamic_rom_ service provides ROM modules that change during the
lifetime of a ROM session according to a timeline. The main purpose of this
service is the automated testing of programs that are able to respond to ROM
module changes, for example configuration changes.
The configuration of the dynamic ROM server contains a '<rom>' sub node per
ROM module provided by the service. Each '<rom>' node hosts a 'name' attribute
and contains a sequence of sub nodes that define the timeline of the ROM
module. The possible sub nodes are:
:'<inline>': The content of the '<inline>' node is assigned to the content
of the ROM module.
:'<sleep>': Sleeps a number of milliseconds as specified via the 'milliseconds'
attribute.
:'<empty>': Removes the ROM module.
At the end of the timeline, it re-starts at the beginning.
Nitpicker GUI server
====================
The nitpicker GUI server has been enhanced to support dynamic screen
resizing. This is needed to let nitpicker respond to screen-resolution
changes, or when using a nested version of nitpicker within a resizable
virtual framebuffer window.
To accommodate Genode's upcoming user-interface concept, we introduced the
notion of a parent-child relationship between nitpicker views. If an existing
view is specified as parent at construction time of a new view, the parent
view's position is taken as the origin of the child view's coordinate space.
This allows for the grouping of views, which can be atomically repositioned by
moving their common parent view. Another use case is the handling of popup
menus in Qt5, which can now be positioned relative to their corresponding
top-level window. The relative position is maintained transparently to Qt when
the top-level window gets repositioned.
Libraries and applications
##########################
Noux runtime for executing Unix software
========================================
Noux plays an increasingly important role for Genode as it allows the use
of the GNU software stack. Even though it already supported a variety of
packages including bash, gcc, binutils, coreutils, make, and vim, some
programs were still limited by Noux' not fully complete POSIX semantics,
in particular with regard to signal handling. For example, it was not
possible to cancel the execution of a long-running process via Control-C.
To overcome those limitations, we enhanced Noux by adding the _kill_ syscall,
reworking the _wait_ and _execve_ syscalls, as well as adding
signal-dispatching code to the Noux libc. Special attention had to be paid to
the preservation of pending signals during the process creation via _fork_ and
_execve_.
The current implementation delivers signals each time a Noux syscall
returns. Signal handlers are executed as part of the normal control flow. This
is in contrast to traditional Unix implementations, which allow the
asynchronous invocation of signal handlers out of band with the regular
program flow. The obvious downside of our solution is that a program that got
stuck in a busy loop (and thereby not issuing any system calls) won't respond
to signals. However, as we regard the Unix interface just as a runtime and not
as the glue that holds the system together, we think that this compromise is
justified to keep the implementation simple and kernel-agnostic. In the worst
case, if a Noux process gets stuck because of such a bug, we certainly can
live with the inconvenience of restarting the corresponding Noux subsystem.
To complement our current activities on the block and file-system levels,
the e2fsprogs-v1.42.9 package as been ported to Noux. To allow the
block-device utilities to operate on Genode's block sessions, we added a new
"block" file system to Noux. Such a block file system can be mounted using a
'<block>' node within the '<fstab>'. By specifying a label attribute, each
block session request can be routed to the proper block session provider:
! <fstab>
! ...
! <dir name="dev">
! <block name="blkdev0" label="block_session_0" />
! </dir>
! ...
! </fstab>
In addition to this file system, support for the DIOCGMEDIASIZE ioctl
request was added. This request is used by FreeBSD and therefore by our
libc to query the size of the block device in bytes.
Qt5 refinements
===============
Our port of Qt5 used to rely on custom versions of synchronization
primitives such as 'QWaitCondition' and 'QMutex'. However, since most of the
usual pthread synchronization functions as relied on by Qt5's regular POSIX
back end have been added to Genode's pthread library by now, we could replace
our custom implementations by Qt5's POSIX version.
Platforms
#########
Execution on bare hardware (base-hw)
====================================
The development of our base-hw kernel platform during this release cycle was
primarily geared towards adding multi-processor support. However, as we
haven't exposed the code to thorough testing yet, we deferred the integration
of this feature for the current release.
We increased the number of usable ARM platforms by adding basic support for
the ODROID XU board.
NOVA microhypervisor
====================
The port of VirtualBox to Genode prompted us to improve the NOVA platform in
the following respects.
NOVA used to omit the propagation of the FPU state of the guest OS to the
virtual machine monitor (VMM) during the world switch between the guest OS and
the VMM. With the Vancouver VMM, which is traditionally used on NOVA, this
omission did not pose any problem because Vancouver would never touch the FPU
state of the guest. So the FPU context of the guest was always preserved
throughout the handling of virtualization events. However, in contrast to the
Vancouver VMM, VirtualBox relies on the propagation of the FPU state between
the guest running in VT-X non-root mode and the guest running within the
VirtualBox recompiler. Without properly propagating the FPU state between both
virtualization back ends, both the guest OS in non-root mode and VirtualBox's
recompiler would corrupt each other's FPU state. After first implementing an
interim solution in our custom version of the kernel, the missing FPU context
propagation had been implemented in the upstream version of NOVA as well.
In contrast to most kernels, NOVA did not allow a thread to yield its current
time slice to another thread. The only way to yield CPU time was to block on
a semaphore or to perform an RPC call. Unfortunately both of those instruments
require the time-receiving threads to explicitly unblock the yielding thread
(by releasing the semaphore or replying to the RPC call). However, there are
situations where the progress of a thread may depend on an external
condition or a side effect produced by another (unknown) thread. One
particular example is the spin lock used to protect (an extremely short)
critical section of Genode's lock metadata. Apparently VirtualBox presented
us with several more use cases for thread-yield semantics. Therefore, we
decided to extend NOVA's kernel interface with a new 'YIELD' opcode to the
'ec_control' system call.

1123
doc/release_notes/14-05.txt Normal file
View File

File diff suppressed because it is too large Load Diff

1136
doc/release_notes/14-08.txt Normal file
View File

File diff suppressed because it is too large Load Diff

1047
doc/release_notes/14-11.txt Normal file
View File

File diff suppressed because it is too large Load Diff

899
doc/release_notes/15-02.txt Normal file
View File

@@ -0,0 +1,899 @@
===============================================
Release notes for the Genode OS Framework 15.02
===============================================
Genode Labs
Genode's [https://genode.org/about/road-map - roadmap] for this year puts a
strong emphasis on the consolidation and cultivation of the existing feature
set. With the first release of the year, version 15.02 pays tribute to this
mission by stepping up to extensive and systematic automated testing. As
a precondition for scaling up Genode's test infrastructure, the release
features a highly modular tool kit for exercising system scenarios on a growing zoo
of test machines. Section [Modular tool kit for automated testing] explains
the new tools in detail. In the spirit of improving the existing feature
set, Genode 15.02 vastly improves the performance and stability of our version of
VirtualBox running on the NOVA microhypervisor, solves long-standing shortcomings
of memory management on machines with a lot of RAM, addresses NOVA-related
scalability limitations, stabilizes our Rump-kernel-based file-system server,
and refines the configuration interface of the Intel wireless driver.
As the most significant new feature, the new version introduces virtualization
support for ARM to our custom base-hw kernel. Section [Virtualization on ARM]
outlines the design and implementation of this feature, which was greatly
inspired by NOVA's virtualization architecture and has been developed over the
time span of more than a year.
With respect to platform support, we are happy to accommodate the upcoming
USB-Armory board, which is a computer in the form factor of a USB
stick especially geared towards security applications. Section
[Support for the USB-Armory board] covers the background and the current
state of this line of work.
Virtualization on ARM
#####################
The ARMv7 architecture of recent processors like Cortex-A7, Cortex-A15, or
Cortex-A17 CPUs support hardware extensions to facilitate virtualization of
guest operating systems. With the current release, we enable the use of these
virtualization extensions in our custom base-hw kernel when running on the
Cortex-A15-based Arndale board.
While integrating ARM's virtualization extension, we aimed to strictly follow
microkernel-construction principles. The primary design is inspired by the
[https://hypervisor.org/ - NOVA OS Virtualization Architecture]. It is based on a
microhypervisor that provides essential microkernel mechanisms along with
basic primitives to switch between virtual machines (VMs). On top of the
microhypervisor, classical OS services are implemented as
ordinary, unprivileged user-level components. Those services can be used by other
applications. Services may be shared between applications or instantiated
separately, according to security and safety needs. Correspondingly,
following the NOVA principles, each VM has its own associated virtual-machine
monitor (VMM) that runs as an unprivileged user-level component. VMM implementations
can range from simple ones that just emulate primary device requirements to highly
complex monitors including sophisticated device models, like VirtualBox. The
NOVA approach allows to decouple the TCB complexity of one VM with respect to
another, as well as with respect to all components not related to
virtualization at all.
Along those lines, we extended the base-hw kernel/core conglomerate with API
extensions that enable user-level VMM components to create and control virtual
machines.
Design
======
The ARM virtualization extensions are based on the so-called security
extensions, commonly known as
[https://genode.org/documentation/articles/trustzone - TrustZone].
The ARM designers did not follow the
Intel approach to split the CPU into a "root" and a "guest" world while having all prior
existing CPU modes available in both worlds. Instead, ARM added a new privilege level
to the non-secure side of TrustZone that sits underneath the ordinary kernel
and userland privilege levels. It is subjected to a hypervisor-like kernel. All
instructions used to prepare a VM's environment have to be executed in this so
called "hyp" mode. In hyp mode, some instructions
differ from their regular behaviour on the kernel-privilege level.
For this reason, prior-existing kernel code cannot simply be reused in
hyp mode without modifications.
The base-hw kernel is meant to execute Genode's core component on bare hardware.
Core, which is an ordinary user-level component, is
linked together with a slim kernel library that is executed in privileged kernel
mode. To enable ARM hardware virtualization, we pushed this approach
even further by executing core in three different privilege levels. Thereby,
core shares the same view on hardware resources and virtual memory across all
levels. A code path is executed on a higher privilege level only if the code
would fail to execute on a lower privilege level.
Following this approach, we were able to keep most of the existing kernel code
with no modifications.
[image avirt_overview]
Genode's ARM kernel (core) runs across all privilege levels
The hypervisor part of core is solely responsible to switch between VMs and the
host system. Therefore, it needs to load/store additional CPU state that
normally remains untouched during context switches of ordinary tasks. It also needs to
configure the VM's guest-physical to host-physical memory translations. Moreover, the
virtualization extensions of the ARMv7 architecture are not related to the CPU
cores only. The interrupt controller and the CPU-local timers are also
virtualization-aware. Therefore, the hypervisor has to load/store state specific
to those devices, too. Nevertheless, the hypervisor merely reloads those
devices. It does not interpret their state.
In contrast to the low-complexity hypervisor, a user-level VMM can be complex
without putting the system's security at risk. It contains potentially complex
device-emulation code and assigns hardware resources such as memory and
interrupts to the VM. The VMM is an ordinary user-level component running
unprivileged. Of course, as a plain user-level component, it is not able to
directly access hardware resources. Hence an interface between VMMs and the
kernel is needed to share the state of a virtual machine. In the past, we faced a similar
problem when building a VMM for our former TrustZone experiments. It was natural
to build upon the available solution and to extend it where necessary. Core
provides a so-called VM service. Each VM corresponds to a session of this
service. The session provides the following extended interface:
:CPU state:
The CPU-state function returns a dataspace containing the virtual machine's
state. The state is initialized by the VMM before bootstrapping the VM, gets updated
by the hypervisor whenever it switches away from the VM, and can be used by
the VMM to interpret the behavior of the guest OS. Moreover, the CPU state can be
updated after the virtual machine monitor emulated instructions
for the VM.
:Exception handler:
The second function is used to register a signal handler that gets informed
whenever the VM produces a virtualization fault.
:Run:
The run function starts or resumes the execution of the VM.
:Pause:
The pause function removes the VM from the kernel's scheduler.
:Attach:
This function attaches a given RAM dataspace to a designated area of the
guest-physical address space.
:Detach:
The detach function invalidates a designated area of the guest-physical
address space.
:Attach_pic: Tells the hypervisor to attach the CPU's virtual interface of the
virtualization-aware interrupt controller to a designated area of the
guest-physical address space.
Implementation
==============
By strictly following the micro-kernel construction principles when integrating the
hypervisor into the base-hw kernel, we reached a minimally invasive solution. In
doing so, we took the time to separate TrustZone-specific code that was formerly
an inherent part of the kernel on ARMv7 platforms. Now, TrustZone- and
virtualization-specific aspects are incorporated into the kernel only if
actually used. The change in complexity of the whole core component expressed in
lines of code is shown in the table below. As can be seen, the additional code in
the root of the trusted computing base when using virtualization is about 700-800
LOC.
Platform | with TrustZone, no VT | TrustZone/VT optional
-----------------------------------------------------------------
hw_arndale | 17970 LOC | 18730 LOC
----------------------------------------------------------------
hw_imx53_qsb | 17900 LOC | 17760 LOC
----------------------------------------------------------------
hw_imx53_qsb_tz | 18260 LOC | 18320 LOC
----------------------------------------------------------------
hw_rpi | 17500 LOC | 17430 LOC
----------------------------------------------------------------
hw_panda | 18040 LOC | 17880 LOC
----------------------------------------------------------------
hw_odroid_xu | 17980 LOC | 18050 LOC
Besides the VM world switch, we enabled support for the so-called "large
physical address extension" (LPAE), which is obligatory when using
virtualization. It allows for addressing a 40-bit instead of only 32-bit physical
address space. Moreover, to execute in hypervisor mode, the bootstrap code of
the kernel had to be set up properly. Hence, when booting on the Arndale board,
the kernel now prepares the non-secure TrustZone world first, and finally leaves the
secure world forever.
To test and showcase the ARM virtualization features integrated in base-hw, we
implemented a minimal, exemplary VMM. It can be found in
_repos/os/src/server/vmm_. The VMM emulates a simplified variant of ARM's
Versatile Express Cortex-A15 development platform. Currently, it only comprises
support for the CPU, the timer, the interrupt controller, and a UART device. It is
written in 1100 lines of C++ in addition to the base Genode libraries. The VMM
is able to boot a vanilla Linux kernel compiled with a slightly modified
standard configuration (no-SMP), and a device tree description stripped down to
the devices provided by the VMM. This release includes an automated run test that
executes the Linux kernel on top of the VMM on Genode. It can be started via:
! make run/vmm
[image avirt_screen]
Three Linux serial consoles running in parallel on top of Genode
Modular tool kit for automated testing
######################################
In
[https://genode.org/documentation/release-notes/13.05#Automated_quality-assurance_testing - Genode version 13.05],
we already introduced comprehensive support for the automated testing of
Genode scenarios. Since then, Genode Labs has significantly widened the scope
of its internal test infrastructure, both in terms of the coverage of the test
scenarios as well as the variety of the used hardware platforms.
The centerpiece of our test infrastructure is the so-called run tool. Steered
by a script (run script), it performs all the steps necessary to test drive
a Genode system scenario. Those steps are:
# *Building* the components of a scenario
# *Configuration* of the init component
# Assembly of the *boot directory*
# Creation of the *boot image*
# *Powering-on* the test machine
# *Loading* of the boot image
# Capturing the *LOG output*
# *Validation* of the scenario behavior
# *Powering-off* the test machine
Each of those steps depends on various parameters such as the
used kernel, the hardware platform used to run the scenario, the
way the test hardware is connected to the test infrastructure
(e.g., UART, AMT, JTAG, network), the way the test hardware is powered or
reseted, or the way of how the scenario is loaded into the test hardware.
Naturally, to accommodate the growing variety of combinations of those
parameters, the complexity of the run tool increased over time.
This growth of complexity prompted us to eventually turn the run tool into a
highly modular and extensible tool kit.
Originally, the run tool consisted of built-in rules that could be
extended and tweaked by a kernel-specific supplement called run environment.
The execution of a run script used to depend on the policies built into
the run tool, the used run environment, and optional configuration
parameters (run opts).
The new run tool kit replaces most of the formerly built-in policies by the
ability to select and configure different modules for the various steps.
The selection and configuration of the modules is expressed in the run-tool
configuration. There exist the following types of modules:
:boot-dir modules:
These modules contain the functionality to populate the boot directory
and are specific to each kernel. It is mandatory to always include the
module corresponding to the used kernel.
_(the available modules are: linux, hw, okl4, fiasco, pistachio, nova,_
_codezero, foc)_
:image modules:
These modules are used to wrap up all components used by the run script
in a specific format and thereby prepare them for execution.
Depending on the used kernel, different formats can be used. With these
modules, the creation of ISO and disk images is also handled.
_(the available modules are: uboot, disk, iso)_
:load modules:
These modules handle the way the components are transfered to the
target system. Depending on the used kernel there are various options
to pass on the components. For example, loading from TFTP or via JTAG is handled
by the modules of this category.
_(the available modules are: tftp, jtag, fastboot)_
:log modules:
These modules handle how the output of a currently executed run script
is captured.
_(the available modules are: qemu, linux, serial, amt)_
:power_on modules:
These modules are used for bringing the target system into a defined
state, e.g., by starting or rebooting the system.
_(the available modules are: qemu, linux, softreset, powerplug, amt)_
:power_off modules:
These modules are used for turning the target system off after the
execution of a run script.
_(the available modules are: powerplug)_
When executing a run script, only one module of each category must be used.
Each module has the form of a script snippet located under the
_tool/run/<step>/_
directory where _<step>_ is a subdirectory named after the module type.
Further instructions about the use of each module (e.g., additional
configuration arguments) can be found in the form of comments inside the
respective script snippets.
Thanks to this modular structure,
the extension of the tool kit comes down to adding a file at the corresponding
module-type subdirectory. This way, custom work flows (such as tunneling JTAG
over SSH) can be accommodated fairly easily.
Usage examples
==============
To execute a run script, a combination of modules may be used. The combination
is controlled via the RUN_OPT variable used by the build framework. Here are a
few common exemplary combinations:
Executing NOVA in Qemu:
!RUN_OPT = --include boot_dir/nova \
! --include power_on/qemu --include log/qemu --include image/iso
Executing NOVA on a real x86 machine using AMT for resetting the target system
and for capturing the serial output while loading the files via TFTP:
!RUN_OPT = --include boot_dir/nova \
! --include power_on/amt --power-on-amt-host 10.23.42.13 \
! --power-on-amt-password 'foo!' \
! --include load/tftp --load-tftp-base-dir /var/lib/tftpboot \
! --load-tftp-offset-dir /x86 \
! --include log/amt --log-amt-host 10.23.42.13 \
! --log-amt-password 'foo!'
Executing Fiasco.OC on a real x86 machine using AMT for resetting, USB serial
for output while loading the files via TFTP:
!RUN_OPT = --include boot_dir/foc \
! --include power_on/amt --amt-host 10.23.42.13 --amt-password 'foo!' \
! --include load/tftp --tftp-base-dir /var/lib/tftpboot \
! --tftp-offset-dir /x86 \
! --include log/serial --log-serial-cmd 'picocom -b 115200 /dev/ttyUSB0'
Executing base-hw on a Raspberry Pi using powerplug to reset the hardware,
JTAG to load the image and USB serial to capture the output:
!RUN_OPT = --include boot_dir/hw \
! --include power_on/powerplug --power-on-powerplug-ip 10.23.42.5 \
! --power-on-powerplug-user admin \
! --power-on-powerplug-password secret \
! --power-on-powerplug-port 1
! --include power_off/powerplug --power-off-powerplug-ip 10.23.42.5 \
! --power-off-powerplug-user admin \
! --power-off-powerplug-password secret \
! --power-off-powerplug-port 1
! --include load/jtag \
! --load-jtag-debugger /usr/share/openocd/scripts/interface/flyswatter2.cfg \
! --load-jtag-board /usr/share/openocd/scripts/interface/raspberrypi.cfg \
! --include log/serial --log-serial-cmd 'picocom -b 115200 /dev/ttyUSB0'
After the run script was executed successfully, the run tool will print the
string 'Run script execution successful.". This message can be used to check
for the successful completion of the run script when doing automated testing.
Meaningful default behaviour
============================
To maintain the ease of use of creating and using a build directory, the
'create_builddir' tool equips a freshly created build directory with a meaningful
default configuration that depends on the selected platform. For example, if
creating a build directory for the Linux base platform, RUN_OPT
is initially defined as
! RUN_OPT = --include boot_dir/linux \
! --include power_on/linux --include log/linux
Low-level OS infrastructure
###########################
Improved management of physical memory
======================================
On machines with a lot of memory, there exist constraints with regard to
the physical address ranges of memory:
* On platforms with a non-uniform memory architecture, subsystems should
preferably use memory that is local to the CPU cores the subsystem is using.
Otherwise the performance is impeded by costly memory accesses to
the memory of remote computing nodes.
* Unless an IOMMU is used, device drivers program physical addresses
into device registers to perform DMA operations. Legacy devices such as
USB UHCI controllers expect a 32-bit address. Consequently, the memory
used as DMA buffers for those devices must not be allocated above 4 GiB.
* When using an IOMMU on NOVA, Genode represents the address space
accessible by devices (by the means of DMA) using a so-called device PD
([https://genode.org/documentation/release-notes/13.02#DMA_protection_via_IOMMU]).
DMA transactions originating from PCI devices are subjected to the virtual
address space of the device PD.
All DMA buffers are identity-mapped with their physical addresses within
the device PD. On 32-bit systems with more than 3 GiB of memory, this
creates a problem. Because the device PD is a regular user-level component, the
upper 1 GiB of its virtual address space is preserved for the kernel. Since
no user-level memory objects can be attached to this
area, the physical address range to be used for DMA buffers is limited
to the lower 3 GiB.
Up to now, Genode components had no way to influence the allocation of
memory with respect to physical address ranges. To solve the problems outlined
above, we extended core's RAM services to take allocation constraints
as session arguments when a RAM session is created. All dataspaces created
from such a session are subjected to the specified constraints. In particular,
this change enables the AHCI/PCI driver to allocate DMA buffers at suitable
physical address ranges.
This innocent looking feature to constrain RAM allocations raises a problem
though: If any component is able to constrain RAM allocations in
arbitrary ways, it would become able to scan the physical address space for
allocated memory by successively opening RAM sessions with the constraints set
to an individual page and observe whether an allocation succeeds or not. Two
conspiring components could use this information to construct a covert storage
channel.
To prevent such an abuse, the init component filters out allocations
constrains from RAM-session requests unless explicitly permitted. The
permission is granted by supplementing the RAM resource assignment of
a component with a new 'constrain_phys' attribute. For example:
! <resource name="RAM" quantum="3M" constrain_phys="yes"/>
Init component
==============
Most of Genode's example scenarios in the form of run scripts support
different platforms. However, as the platform details vary, the run scripts
have to tweak the configuration of the init component according to the
features of the platform.
For example, when declaring an explicit route to a framebuffer driver named
"fb_drv", the run script won't work on Linux because on this platform, the
framebuffer driver is called "fb_sdl".
Another example is the role of the USB driver. Depending on the platform, the
USB driver is an input driver, a block driver, a networking driver, or a
combination of those.
Consequently, run scripts with support
for a great variety of platforms tend to become convoluted with
platform-specific conditionals.
To counter this problem, we enhanced init to support aliases for component
names. By defining the following aliases in the init configuration
! <alias name="nic_drv" child="usb_drv"/>
! <alias name="input_drv" child="usb_drv"/>
! <alias name="block_drv" child="usb_drv"/>
the USB driver becomes reachable for session requests routed to either "usb_drv",
"nic_drv", "input_drv", and "block_drv". Consequently, the routing
configuration of components that use either of those drivers does no longer
depend on any platform-intrinsic knowledge.
RTC session interface
=====================
Until now, the RTC session interface used an integer to return the current
time. Although this is preferable when performing time-related
calculations, a structured representation is more convenient to use, i.e., if
the whole purpose is showing the current time. This interface change is only
visible to components that use the RTC session directly.
Since the current OS API of Genode lacks time-related functions, most users
end up using the libc, which already converts the structured time stamp
internally, or provide their own time related functions.
Update of rump-kernel-based file systems
========================================
We updated the rump-kernel support to a newer rump-kernel version (as of mid of
January 2015). This way, Genode is able to benefit from upstream stability
improvements related to the memory management. Furthermore, we revised the
Genode backend to allow the rump_fs server to cope well with a large amount of
memory assigned to it. The latter is useful to utilize the block cache of the
NetBSD kernel.
Libraries and applications
##########################
As a stepping stone in the
[https://github.com/genodelabs/genode/issues/1399 - forthcoming community effort]
to bring the Nix package manager to Genode, ports of libbz2 and sqlite have
been added to the _repos/libports/_ repository.
Runtime environments
####################
VirtualBox on NOVA
==================
Whereas our previous efforts to run VirtualBox on Genode/NOVA were mostly
concerned with enabling principal functionality and with the addition of
features, we took the release cycle of Genode 15.02 as a chance to focus
on performance and stability improvements.
:Performance:
Our goal with VirtualBox on NOVA is to achieve a user experience
comparable to running VirtualBox on Linux. Our initial port of VirtualBox used
to cut a lot of corners with regards to performance and timing accuracy
because we had to concentrate on more fundamental issues of the porting
work first. Now, with the feature set settled, it was time to revisit
and solidify our interim solutions.
The first category of performance improvements is the handling of timing,
and virtual guest time in particular. In our original version,
we could observe a substantial drift of the guest time compared to the host time.
The drift is not merely inconvenient but may even irritate the guest OS
because it violates its assumptions about the behaviour of certain virtual devices.
The drift was caused by basing the timing on a simple jiffies counter
that was incremented by a thread after sleeping for a fixed period. Even
though the thread almost never executes, there is still a chance that it gets
preempted by the kernel and resumed only after the time slices of
concurrently running threads have elapsed. This can take tens of milliseconds.
During this time, the jiffies counter remains unchanged. We could
significantly reduce the drift by basing the timing on absolute time values
requested from the timer driver. Depending on the used guest OS, however,
there is still a residual inaccuracy left, which is subject to ongoing
investigations.
The second type of improvements is related to the handling of virtual
interrupts. In its original habitat, VirtualBox relies on so-called
external-interrupt virtualization events. If a host interrupt occurs while the
virtual machine is active, the virtualization event is forwarded by the
VirtualBox hypervisor to the virtual machine monitor (VMM).
On NOVA, however, the kernel does not propagate this
condition to the user-level VMM because the occurrence of host interrupts should
be of no matter to the VMM. In the event of a host interrupt, NOVA takes
a normal scheduling decision (eventually activating the user-level device driver
the interrupt belongs to) and leaves the virtual CPU (vCPU) in a runnable
state - to be rescheduled later. Once the interrupt is handled, the vCPU gets
resumed. The VMM remains out of the loop. Because the update of the VirtualBox
device models ultimately relies on the delivery of external-interrupt
virtualization events, the lack of this kind of event introduced huge delays
with respect to the update of device models and the injection of virtual
interrupts. We solved this problem by exploiting a VirtualBox-internal
mechanism called POKE. By setting the so-called POKE flag, an I/O thread is
able to express its wish to force the virtual machine into the VMM. We only
needed to find the right spots to set the POKE flag.
Another performance-related optimization is the caching of RTC time
information inside VirtualBox. The original version of the gettimeofday
function used by VirtualBox contacted the RTC server for obtaining the
wall-clock time on each call. After the update to VirtualBox 4.3, the rate of those
calls increased significantly. To reduce the costs of these calls, our
new version of gettimeofday combines infrequent calls to the RTC driver
with a component-local time source based on the jiffies mechanism mentioned above.
With these optimizations in place,
simple benchmarks like measuring the boot time of Window 7 or the time of
compiling Genode within a Debian VM suggest that our version of VirtualBox
has reached a performance that is roughly on par with the Linux version.
:Stability:
Since the upgrade to VirtualBox 4.3.16 in release 14.11, we fixed several
regression issues caused by the upgrade. Beside that, we completed the
support to route serial output of guests to Genode, lifted the restriction
to use just one fixed VESA mode, and enabled support for 32-bit Windows 8
guests on 64-bit Genode/NOVA. The 64-bit host restriction stems from
the fact that Windows 8 requires support for the non-executable bit (NX)
feature of page tables. The 32-bit version of the NOVA kernel does not leverage
the physical address extension (PAE) feature, which is a pre-requisite for
using NX on 32-bit.
In the course of the adaptation, our port of VirtualBox now evaluates the
PAE and HardwareVirtExUX XML tags of .vbox files:
!<VirtualBox xmlns=...>
! <Machine uuid=...>
! <Hardware ..>
! <CPU ...>
! <HardwareVirtExUX enabled="true"/>
! <PAE enabled="true"/>
! ...
The PAE tag specifies whether to report PAE capabilities to the guest
or not. The HardwareVirtExUx tag is used by our port to decide whether to stay
for non-paged x86 modes in Virtualbox's recompiler (REM) or not. Until now, we used REM
to emulate execution when the guest was running in real mode and protected mode
with paging disabled. However, newer Intel machines support the unrestricted guest
feature, which makes the usage of REM in non-paged modes not strictly
necessary anymore. Setting the HardwareVirtExUx tag to false accommodates
older machines with no support for the unrestricted-guest feature.
Device drivers
##############
iPXE-based network drivers
==========================
We enabled and tested the driver with Intel I218-LM and I218-V PCI devices.
Intel wireless stack
====================
In this release, several small issues regarding the wireless stack are fixed.
From now on, the driver only probes devices on the PCI bus that correspond to
the PCI_CLASS_NETWORK_OTHER device class. Prior to that, the driver probed all
devices attached to the bus resulting in problems with other devices, e.g.
the GPU, when accessing their extended PCI config space.
Since the driver uses cooperative scheduling internally, it must never block
or, in case it blocks, must schedule another task. Various sleep functions
lacked this scheduling call and are now fixed. Furthermore, a bug in the timer
implementation has been corrected, which caused the scheduling of wrong timeouts.
In addition to these fixes, patches for enabling the support for
Intel 7260 cards were incorporated.
Up to now, the configuration of the wireless driver was rather inconvenient because
it did not export any information to the system. The driver now creates two
distinct reports to communicate its state and information about the wireless
infrastructure to other components. The first one is a list of all available
access points. The following exemplary report shows its structure:
!<wlan_accesspoints>
! <accesspoint ssid="skynet" bssid="00:01:02:03:04:05" quality="40"/>
! <accesspoint ssid="foobar" bssid="01:02:03:04:05:06" quality="70" protection="WPA-PSK"/>
! <accesspoint ssid="foobar" bssid="01:02:03:04:05:07" quality="10" protection="WPA-PSK"/>
!</wlan_accesspoints>
Each '<accesspoint>' node has attributes that contain the SSID and the BSSID
of the access point as well as the link quality (signal strength). These
attributes are mandatory. If the network is protected, the node will also
have an attribute describing the type of protection in addition.
The second report provides information about the state of the connection
with the currently associated access point:
!<wlan_state>
! <accesspoint ssid="foobar" bssid="01:02:03:04:05:06" quality="70"
! protection="WPA-PSK" state="connected"/>
!</wlan_state>
Valid state values are 'connected', 'disconnected', 'connecting' and
'disconnecting'.
The driver obtains its configuration via a ROM module. This ROM
module contains the selected access point and can be updated during runtime.
To connect to an access point, a configuration like the following is used:
!<selected_accesspoint ssid="foobar" bssid="01:02:03:04:05:06"
! protection="WPA-PSK" psk="foobar123!"/>
To disconnect from an access point, an empty configuration can be set:
!<selected_accesspoint/>
For now, the prevalent WPA/WPA2 protection using a pre-shared key is supported.
Improved UART driver for Exynos5
================================
The UART driver for the Exynos5 SoC has been enhanced by enabling the RX
channel. This improvement was motivated by automated tests, where a run script
needs to interact with some component via a terminal connection.
Touchscreen support
===================
We enabled support of Wacom USB touchscreen devices via dde_linux - a port of
Linux USB driver to Genode. In order to make touchscreen coordinates
usable by Genode's input services, they must be calibrated
to screen-absolute coordinates. The screen resolution is not determined
automatically by the USB driver. It can, however, be configured as a sub
node of the '<hid>' XML tag of the USB driver's configuration:
!<start name="usb_drv">
! ...
! <config uhci=... ohci=... xhci=...>
! <hid>
! <screen width="1024" height="768"/>
! </hid>
! ...
USB session interface
=====================
We enhanced our USB driver with the support of remote USB sessions. This
feature makes it possible to implement USB-device drivers outside the USB
server using a native Genode API. The new USB session can be found under
_repos/os/include/usb_session_ and can be used to communicate with the USB
server, which merely acts as a host controller and HUB driver in this scenario.
Under _repos/os/include/usb_, there are a number of convenience
and wrapper functions that operate directly on top of a USB session. These
functions are meant to ease the task of USB-device-driver programming by hiding
most of the USB session management, like packet-stream handling.
We also added a USB terminal server, which exposes a Genode terminal session to
its clients and drives the popular PL2303 USB to UART adapters using the new
USB-session interface.
A practical use case for this component is the transmission of logging data on
systems where neither UART, AMT, nor JTAG are available. A run script
showcasing this feature can be found at _repos/dde_linux/run/usb_terminal.run_.
RTC proxy driver for Linux
==========================
There are a handful of run scripts that depend on the RTC service. So far,
it was not possible to run these tests on Linux due to the lack of an RTC
driver on this platform. To address this problem, we created a proxy driver
that uses the time() system call to provide a
reasonable base period on Linux.
Platforms
#########
Execution on bare hardware (base-hw)
====================================
Support for the USB-Armory board
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
With [https://www.crowdsupply.com/inverse-path/usb-armory - USB Armory],
there is an intriguing hardware platform for Genode on the horizon.
In short, USB Armory is a computer in the form factor of a USB
stick. It is meant for security applications such as VPNs,
authentication tokens, and encrypted storage. It is based on the
FreeScale i.MX53 SoC, which is well supported by Genode, i.e.,
Genode can be used as secure-world OS besides Linux running in the
normal world.
Apart from introducing a novel form factor, this project is
interesting because it strives to be an 100% open platform, which
includes hardware, software, and firmware. This motivated us to
bring Genode to this platform.
The underlying idea is to facilitate
[https://genode.org/documentation/articles/trustzone - ARM TrustZone] to
use Genode as a companion to a Linux-based OS on the platform.
Whereas Linux would run in the normal world of TrustZone, Genode runs
in the secure world. With Linux, the normal world will control the
communication over USB and provide a familiar environment to implement
USB-Armory applications. However, security-critical functions and data like
cryptographic keys will reside exclusively in the secure world. Even in
the event that Linux gets compromised, the credentials of the user
will stay protected.
The support of the USB Armory platform was added in two steps:
First, we enabled our base-hw kernel to run as TrustZone monitor with
Genode on the "secure side". Since the USB Armory is based on the
FreeScale i.MX53 SoC, which Genode already supported, this step went
relatively straight-forward.
Second, we enabled a recent version of the Linux kernel (3.18) to run in the
normal world. The normal world is supervised by a user-level Genode component
called tz_vmm (TrustZone Virtual Machine Monitor). The tz_vmm is, among
others, responsible for providing startup and hardware information to the
non-secure guest. The Linux kernel version we used previously as TrustZone
guest on i.MX53 boards expected this information to be communicated via
so-called ATAGs. The new version, however, expects this to be done via a
device tree blob. As a consequence, the tz_vmm had to be adapted to properly
load this blob into the non-secure RAM. The original USB-Armory device tree
was modified to blind out the RAM regions that get protected by the TrustZone
hardware. This way, Linux won't attempt to access them. Furthermore,
to keep basic user interaction simple, our device tree tells Linux to use the
same non-secure UART as Genode for console I/O.
The kernel itself received some modifications, for two reasons. First,
we don't want Linux to rely on resources that are protected to keep
the secure world secure. This is why the driver for the interrupt controller
that originally made use of the TrustZone interrupt configuration, had to be
adapted. Second, to prevent Linux from disturbing Genode activities, we
disabled most of the dynamic clock and power management as it may sporadically
gear down or even disable hardware that Genode relies on. Furthermore, we
disabled the Linux drivers for I2C interfaces and the GPIO configuration as
these are reserved for Genode.
IPC helping
~~~~~~~~~~~
In traditional L4 microkernels, scheduling parameters (like time-slice
length and priority) used to be bound to threads. Usually, those parameters
are defined at thread creation time. The initial version
of base-hw followed this traditional approach. However, it has a few problems:
* For most threads, the proper *choice of scheduling parameters* is very
difficult if not impossible. For example, the CPU-time demands of a
server thread may depend on the usage patterns of its clients. Most
theoretical work in the domain of scheduling presumes the knowledge of
job lengths in advance of computing a schedule. But in practice and in
particular in general-purpose computing, job lengths are hardly known a priori.
As a consequence, in most scenarios, scheduling parameters are
set to default values.
* With each thread being represented as an independent schedulable entity,
the kernel has to take a scheduling decision each time a thread performs an
IPC call because the calling thread gets blocked and the called thread
may get unblocked. In a microkernel-based system, those events occur at a
much higher rate than the duration of typical time slices, which puts the
scheduler in a *performance-critical* position.
* Regarding IPC calls, a synchronous flow of control along IPC call chains is
desired. Ideally, an IPC call should have the same characteristics as
a function call with respect to scheduling. When a client thread performs an
IPC call, it expects the server to immediately become active to
handle the request. But if the kernel treats each thread independently,
it may pick any other thread and thereby introduce *high latencies* into
IPC operations.
To counter those problems, the NOVA microhypervisor introduced a new approach
that decouples scheduling parameters from threads. Instead of selecting
threads for execution, the scheduler selects so-called scheduling contexts.
For a selected scheduling context, the kernel dynamically determines a
thread to execute by taking IPC relationships into account. When a thread
performs an IPC, the thread's scheduling context will be used to execute
the called server. In principle, a server does not need CPU time on its own
but always works with CPU resources provided by clients.
The new version of the base-hw kernel adapts NOVA's approach with slight
modifications. Each thread owns exactly one scheduling context for its entire
lifetime. However, by the means of "helping" during an IPC call, the caller
lends its scheduling context to the callee. Even if the callee is still busy
and cannot handle the IPC request right away, the caller helps because it
wants the callee to become available for its request as soon as
possible. Consequently, a thread has potentially many scheduling contexts at
its disposal, its own scheduling context plus all scheduling contexts
provisioned by helpers. This works transitively.
Purged outdated platforms
~~~~~~~~~~~~~~~~~~~~~~~~~
We removed the support for two stale platforms that remained unused for
more than a year, namely FreeScale i.MX31 and the TrustZone variant
of the Coretile Versatile Express board.
NOVA
====
On Genode/NOVA, we used to employ one pager thread in core for each thread
in the system. We were forced to do so because not every page
fault can be resolved immediately. In some situations, core asynchronously
propagates the fault to an external component for the resolution.
In the meantime, the
pager thread leaves the page fault unanswered. Unfortunately, the kernel
provides no mechanism to support this scenario besides just blocking the
pager thread using a semaphore. This, in turn, means that the pager thread is not
available for other page-fault requests. Ultimately, we had to setup a
dedicated pager per thread.
This implementation has the downside of "wasting" memory for a lot of
pager threads. Moreover, it becomes a denial-of-service vector as soon as more
threads get created than core can accommodate. The number of threads is
limited per address space - also for core - by the size of Genode's context
area, which typically means 256 threads.
To avoid the downsides mentioned, we extended the NOVA IPC reply syscall to
specify an optional semaphore capability. The NOVA kernel validates the
capability and blocks the faulting thread in the semaphore. The faulted thread
remains blocked even after the pager has replied to the fault message. But
the pager immediately becomes available for other
page-fault requests. With this change, it suffices to maintain only one pager
thread per CPU for all client threads.
The benefits are manifold. First, the base-nova implementation converges more
closely to other Genode base platforms. Second, core can not run out of threads
anymore as the number of threads in core is fixed for a given setup. And the
third benefit is that the helping mechanism of NOVA can be leveraged for
concurrently faulting threads.
Build system and tools
######################
Tools for convenient handling of port contrib directories
=========================================================
We supplemented our tools for the ports mechanism with two convenient
scripts:
:_tool/ports/shortcut_:
Creates a symbolic link from _contrib/<port-name>-<hash>_ to
_contrib/<port-name>_. This is useful when working on the third-party
code contained in the _contrib_ directory.
:_tool/ports/current_:
Prints the current contrib directory of a port. When switching
branches back and forth, the hash of the used port might change.
The script provides a shortcut to looking up the hash file for a
specific port within the repositories and printing its content.

1216
doc/release_notes/15-05.txt Normal file
View File

File diff suppressed because it is too large Load Diff

791
doc/release_notes/15-08.txt Normal file
View File

@@ -0,0 +1,791 @@
===============================================
Release notes for the Genode OS Framework 15.08
===============================================
Genode Labs
The version 15.08 marks the beginning of Genode as day-to-day OS as one of the
project's core developers switched to using Genode/NOVA on his machine,
stressing the OS infrastructure we created over the course of the last seven
years. Thanks to components like VirtualBox, the Noux runtime for GNU software,
the Linux wireless stack and Rump-kernel-based file systems, the transition
went actually much smoother than expected. So other members of the team plan
to follow soon. Section [Genode as day-to-day operating system] gives an
overview of the taken approach. Genode's use as general-purpose OS provided
the incentive for most of the improvements featured by the current release,
starting with the addressing of the long-standing kernel-memory management
deficiencies of the NOVA kernel (Section [NOVA kernel-resource management]),
over enhancements of Genode's tracing and file-system facilities, to vast
improvements of the guest-host integration of VirtualBox when running on
Genode.
The release is accompanied with a second line of work led by our friends
at Codelabs: Enabling Genode to run on top of their Muen separation
kernel as described in Section [Genode on top of the Muen Separation Kernel].
Muen is a low-complexity kernel for the 64-bit x86 architecture that
statically partitions the machine into multiple domains. In contrast to
microkernels like the ones already supported by Genode, the assignment
of physical resources (such as memory, CPU time, and devices) happens at
system-integration time. Since an isolation kernel does not have to deal
with dynamic resource management at runtime, it is less complex than
a general-purpose microkernel. This makes it relatively easy to reason about
its strong isolation properties, which, in turn, makes it attractive for
high-assurance computing. With Genode being able to run within a Muen
domain, the rich component infrastructure of Genode can be combined with
the strong isolation guarantees of Muen.
Genode on top of the Muen Separation Kernel
###########################################
_This section was written by Adrian-Ken Rueegsegger and Reto Buerki who_
_conducted the described line of work independent from Genode Labs._
After completing our x86_64 port of the Genode base-hw kernel, which was
featured in the
[https://genode.org/documentation/release-notes/15.05#Principal_support_for_the_64-bit_x86_architecture - previous release (15.05)],
we immediately started working on our main goal: running a Genode system as
guest on the Muen Separation Kernel (SK). This would enable the Muen platform
to benefit from the rich ecosystem of Genode.
For those who have not read the 15.05 Genode release notes, [https://muen.sk - Muen]
is an Open-Source microkernel, which uses the [https://spark-2014.org/ - SPARK]
programming language to enable light-weight formal methods for high assurance.
The 64-bit x86 kernel, currently consisting of a little over 5'000 LOC, makes
extensive use of the latest Intel virtualization features and has been formally
proven to contain no runtime errors at the source-code level.
The new 'hw_x86_64_muen' platform, as the name implies, extends the 'hw_x86_64'
base-hw kernel by replacing the PIC and timer drivers with paravirtualized
variants.
In contrast to other kernels supported by Genode, the architecture with Muen is
different in the sense that the entire 'hw_x86_64_muen' Genode system runs as
guest VM in VMX non-root mode on the SK. From the perspective of Muen, Genode
is executed on top of the kernel like any other guest OS without special
privileges.
[image muen_system_overview]
Genode running on top of the Muen Separation Kernel alongside other subjects
This loose coupling of Muen and Genode base-hw enables the robust combination
of a static, low-complexity SK with a feature-rich and extensive OS framework.
The result is a flexible platform for the construction of component-based
high-assurance systems.
People interested in giving the 'hw_x86_64_muen' platform a spin can find a
small tutorial at _repos/base-hw/doc/x86_64_muen.txt_.
NOVA kernel-resource management
###############################
For several years, the NOVA kernel has served as Genode's primary base
platform on x86. The main reasons for this choice are: the kernel provides -
among the supported x86 kernels - the richest feature set like the support of
IOMMUs, virtualization, and SMP. It also offers a clean design and a stable
kernel interface. The available kernel-interface specification and the
readable and modern source base are a pleasure to work with. Hence, Genode
Labs is able to fully commit to the maintenance and further evolution of this
kernel.
Nevertheless, since the beginning, the vanilla kernel lacks one essential
feature to reliably host Genode as user-land, namely the proper management of
the memory used by the kernel itself (in short kernel-memory management). In
the past, we already extended the kernel to free up kernel resources when
destroying kernel objects, e.g., protection domains and page-tables, threads,
semaphores, and portals. Still, on Genode/NOVA, a component may trigger
arbitrary kernel-memory consumption during RPC by delegating memory,
capabilities, or by creating other components via Genode's core component. If
the kernel memory gets depleted, the kernel panics with an "Out of memory"
message and the entire Genode scenario stops.
In principal, the consumption of kernel memory can be deliberately provoked by
a misbehaving (greedy) component. But also during the regular day-to-day usage
of Genode, can such a situation occur when the system is used in a highly
dynamic fashion. For example, compiling and linking source code within the
noux environment constantly creates and destroys protection domains, threads,
and memory mappings. Our nightly test of compiling Genode within noux triggers
this condition every once in a while.
The main issue here is that the consumption of kernel memory is not accounted
by Genode. The kernel interface does not support such a feature. Kernels like
seL4 as well as Genode's custom base-hw kernel show how this problem can be
solved.
To improve the current situation - where the overall kernel memory is a fixed
amount - we extended NOVA in the following ways: First, the NOVA kernel
accounts any kernel memory consumption per protection domain. Second, each
process has a limited amount of kernel-memory quota it can use. Last, the
kernel detects when the quota limit of a protection domain is reached.
If the third condition occurs, the kernel stops the offending thread and
(optionally) notifies a handler thread. This so called out-of-memory (OOM)
handler thread receives information about the current situation and may
respond to it in the following ways:
* Stop the thread of the depleted protection domain, or
* Transfer kernel-memory quota between protection domains (upgrading the limit
if desired), or
* Free up kernel memory if possible, e.g., revoke memory delegations, which
can be re-created.
We implemented the steps above inside the NOVA kernel and extended Genode's
core component to handle such OOM situations. All system calls beside the IPC
call/reply may now return an error code upon depletion of the quota. Most of
these system calls can solely be performed by core and are handled inside
core's NOVA-specific platform code.
In the case of IPC call/reply operations, we desired to handle OOM cases
transparently to Genode user-level components. Therefore, each thread in
Genode/NOVA now gets constructed with an OOM IPC portal attached. This portal
is served by the pager thread in core and is traversed on OOM occurrences
during IPC operations. If a pager thread receives such an OOM IPC, it decodes
the involved IPC sender and IPC receiver and locates the appropriate
core-internal paging objects. The currently implemented out-of-memory policy
tries to upgrade the quota. If this is not possible, an attempt to revoke
memory mappings from the OOM-causing protection domain is made. This
implicitly frees-up some kernel memory (e.g., mapping nodes). If none of the
responses suffices, the handler stops the OOM-causing thread and writes a
message to the system log.
The current policy implementation constitutes a rather rough heuristic, which
may not suffice under all circumstances. In the future, we would like to
specify a distinct policy per component, e.g. depending on prior known memory
usage patterns. For example, some components follow well-known usage patterns
and therefore a fixed upper quota limit can be specified. Other components are
highly dynamic and desire quota upgrades on demand. There are many more
combinations imaginable.
Our current plan is to collect more experience over the next months with this
new kernel mechanism. Based on our observations, we may externalize such
policy decisions and possibly make them configurable per component.
The current implementation however, already avoids the situation that the
kernel goes out of service if a single component misbehaves
kernel-memory-wise.
Genode as day-to-day operating system
#####################################
At the beginning of June, Genode reached the probably most symbolic milestone
in the project's history: Norman - one of the core developers - replaced his
Linux-based working environment with a Genode-based system. This system is
composed of the following ingredients:
[image turmvilla_scenario]
The machine used is a Lenovo Thinkpad X201. We settled on this five-year-old
machine for several reasons. First, it is a very solid platform with a nice
form factor. Second, it features Intel's AMT (Active Management Technology),
which is handy to obtain low-level system logs in the case something goes
wrong. Third, refurbished machines of this type can be obtained for as little
as 200 EUR. Finally, an older machine reinforces the need for good performance
of the operating system. So it creates a natural incentive for Norman to find
and address performance bottlenecks.
Our modified version of the NOVA microhypervisor is the used kernel.
The user interface is based on our custom GUI stack including the nitpicker
GUI server as well as the window manager and its companion components
(decorator, layouter, pointer) we introduced in
[https://genode.org/documentation/release-notes/14.08#New_GUI_architecture - version 14.08].
The display is driven by the VESA driver. User input is handled by the PS/2
driver for handling the laptop keyboard and trackpoint, and the USB driver for
handling an externally connected keyboard and mouse.
Network connectivity is provided by our port of the Intel Wireless stack that
we introduced with the version
[https://genode.org/documentation/release-notes/14.11#Intel_wireless_stack - 14.11].
Our custom AHCI driver provides access to the physical hard disk. File-system
access is provided by our
[https://genode.org/documentation/release-notes/14.02#NetBSD_file_systems_using_rump_kernels - Rump-kernel-based file-system server].
A simple Genode shell called CLI monitor allows the user to start and kill
subsystems dynamically. Initially, the two most important subsystems are
VirtualBox and Noux.
VirtualBox executes a GNU/Linux-based guest OS that we refer to as "rich OS".
The rich OS serves as a migration path from GNU/Linux to Genode. It is used
for all tasks that cannot be accomplished directly on Genode yet. At the
beginning of the transition, the daily routine still very much depends on the
rich OS. By moving more and more functionality over to the Genode world, we
will eventually be able to make the rich OS obsolete step by step. Thanks to
VirtualBox' excellent host-guest-integration features, the VirtualBox window
can be dynamically resized and the guest mouse cursor integrates seamlessly
with Genode's pointer. VirtualBox is directly connected to the wireless
network driver. So common applications like Firefox can be used.
The noux runtime allows us to use command-line-based GNU software directly on
Genode. Coreutils and Bash are used for managing files. Vim is used for
editing files. Unlike the rich OS, the noux environment has access to the
Genode partition of the hard disk. In particular, it can be used to update the
Genode system. It has access to a number of pseudo files that contain status
information of the underlying components, e.g., the list of wireless access
points. Furthermore, it has limited access to the configuration interfaces of
the base components. For example, it can point the wireless driver to the
access point to use, or change the configuration of the nitpicker GUI server
at runtime.
As a bridge between the rich OS and the Genode world, we combine VirtualBox'
shared-folder mechanism with Genode's VFS infrastructure. The shared folder is
represented by a dedicated instance of a RAM file system, which is mounted in
both the VFS of VirtualBox and the VFS of noux.
As evidenced by Norman's use since June, the described system setup is
sufficient to be productive. So other members of the Genode team plan to
follow in his footsteps soon. At the same time, the continued use of the
system from day to day revealed a number of shortcomings, performance
limitations, and rough edges, which we eventually eliminated. It goes without
saying that this is an ongoing effort. Eating our own dog food forces us to
address the right issues to make the daily life more comfortable.
Feature-wise the switch to Genode motivated three developments, namely the
enhancement of Genode's CLI monitor, the improvement of the window manager,
and the creation of a CPU-load monitoring tool.
Interactive management of subsystem configurations
==================================================
The original version of CLI monitor obtained the configuration data of its
subsystems at start time via the Genode::config mechanism. But for managing
complex scenarios, the config node becomes very complex. Hence, it is
preferable to have a distinct file for each subsystem configuration.
The new version of CLI monitor scans the directory '/subsystems' for files
ending with ".subsystem". Each file has the same syntax as the formerly used
subsystem nodes. This change has the welcome implication that subsystem
configurations can be changed during the runtime of the CLI monitor, e.g., by
using a concurrently running instance of noux with access to the _subsystems/_
directory. This procedure has become an essential part of the daily work flow
as it enables the interactive evolution of the Genode system.
Window-management improvements
==============================
To make the window manager more flexible while reducing its complexity at the
same time, we removed the formerly built-in policy hosting the decorator and
layout components as children of the window manager. Those components are no
longer child components but siblings. The relationship of the components is
now solely expressed by the configuration of their common parent, i.e., init.
This change clears the way to dynamically replace those components during
runtime (e.g., switching between different decorators).
To improve the usability of the windowed GUI, we enabled the layouter to
raise windows on click and to let the keyboard focus follow the pointer.
Furthermore, the window manager, the decorator, and the floating window
layouter became able to propagate the usage of an alpha channel from the
client application to the decorator. This way, the decorator can paint the
decoration elements behind the affected windows, which would otherwise be
skipped. Consequently, partially transparent windows can be properly displayed.
CPU-load monitoring
===================
During daily system use, we started to wish to know in detail where the CPU
cycles are spent. For example, the access of a file by the rich OS involves
several components, including the guest OS itself, VirtualBox, rump_fs (file
system), part_blk (partition access), ahci_drv (SATA device access), core, and
NOVA. Investigating performance issues requires a holistic view of all those
components. For this reason, we enhanced our existing tracing infrastructure
(Section [Enhanced tracing facilities]) to allow the creation of CPU-load
monitoring tools. The first tool in this category is the graphical CPU-load
monitor located at _gems/app/cpu_load_display/_, which displays a timeline of
the CPU load where each thread is depicted with a different color. Thanks to
this tool, we have become able to explore performance issues in an interactive
way. In particular, it helped us to identify and resolve a long-standing
inaccuracy problem in our low-level timer service.
Base framework and low-level OS infrastructure
##############################################
Improved audio support
======================
In the previous release, we replaced our old audio driver with a new one that
provided the same audio-out session interface. Complementing the audio-out
session, we are now introducing a new audio-in session interface that can be
used to record audio frames. It is modeled after the audio-out interface in
the way how it handles the communication between the client and the server. It
uses shared memory in the form of the Audio_in::Stream to transport the frames
between the components. A server component captures frames and puts them into
a packet queue, which is embedded in the Audio_in::Stream. The server
allocates packets from this queue to store the recorded audio frames. If the
queue is already full, the server will override already allocated packets and
will notify the client by submitting an 'overrun' signal. The client has to
cope with this situation, e.g., by consuming packets more frequently. A client
can install a signal handler to respond to a progress signal, which is sent by
the server when a new Audio_in::Packet has been submitted to the packet queue.
For now, all audio-in server components only support one channel (left)
although the audio-in session interface principally supports multiple
channels.
The _dde_bsd_ audio_drv is the first and currently only audio driver component
that was extended to provide the audio-in session. To express this fact, the
driver was renamed from _audio_out_drv_ to _audio_drv_. In contrast to its
playback functionality, which is enabled by default, recording has to be
enabled explicitly by setting the configuration attribute 'recording' to
'yes'. If the need arises, playback may be disabled by setting 'playback' to
'no'. In addition, it is now possible to configure the driver by adjusting the
mixer in the driver's configuration node. For the time being, the interface as
employed by the original OpenBSD mixer utility is used.
The following snippet shows how to enable and configure recording on a
Thinkpad X220 where the headset instead of the internal microphone is used as
source:
! <start name="audio_drv">
! <resource name="RAM" quantum="8M"/>
! <provides>
! <service name="Audio_out"/>
! <service name="Audio_in"/>
! </provides>
! <config recording="yes">
! <mixer field="outputs.master" value="255"/>
! <mixer field="record.adc-0:1_source" value="sel2"/>
! <mixer field="record.adc-0:1" value="255"/>
! </config>
! </start>
In addition to selecting the recording source, the playback as well as the
recording volume are raised to the maximum. Information about all available
mixers and settings in general may be obtained by specifying the 'verbose'
attribute in the config node.
The enriched driver is accompanied by a simple monitor application, which
directly plays back all recorded audio frames and shows how to use the
audio-in session. It can be tested by executing the
_repos/dde_bsd/run/audio_in.run_ run script.
There are also changes to the audio-out session itself. The length of a period
was reduced from 2048 to 512 samples to accommodate for a lower latency when
mixing audio-out packets. A method for invalidating all packets in the queue
was also added.
File-system infrastructure
==========================
Unlike traditional operating systems that rely on a global name space for
files, each Genode component has a distinct view on files. Many low-level
components do not even have the notion of files. Whereas traditional operating
systems rely on a virtual file system (VFS) implemented in the OS kernel,
Genode's VFS has the form of a library that can optionally be linked to a
component. The implementation of this library originated from the noux runtime
introduced in version
[https://genode.org/documentation/release-notes/11.02#Noux_-_an_execution_environment_for_the_GNU_userland - 11.02],
and was later integrated into our C runtime in version
[https://genode.org/documentation/release-notes/14.05#Per-process_virtual_file_systems - 14.05].
With the current release, we take the VFS a step further by making it
available to components without a C runtime. Thereby, low-complexity
security-sensitive components such as CLI monitor become able to benefit from
the powerful VFS infrastructure.
The VFS itself received a welcome improvement in the form of private RAM file
systems. A need for process-local storage motivated a conversion of the
existing ram_fs server component to an embeddable VFS file system. This
addition to the set of VFS plugins enables components to use temporary file
systems without relying on the resources of an external component.
Unified networking components
=============================
Having had a good experience with our Block::Driver implementation, which
wraps the block-session interface and takes care of the packet-stream
handling, thus easing the implementation of driver and other block components,
we observed that this approach did not provide enough flexibility for
NIC-session servers. For example, NIC servers are bi-directional and when a
network packet arrives the server has to make sure that there are enough
resources available to dispatch the network packet to the client. This has to
be done because the server must never block, e.g., by waiting for allocations
to succeed or for an empty spot in the packet queue of a client. Therefore,
such a non-blocking NIC server needs to validate all preconditions for
dispatching the packet in advance and, if they cannot be met, drop the network
packet.
In order to implement this kind of behavior, NIC-session servers must have
direct access to the actual NIC session. For this reason, we removed the
Nic::Driver interface from Genode and added a Nic::Session_component that
offers common basic packet-stream-signal dispatch functionality. Servers may
now inherit from this component and implement their own policy.
We adjusted all servers that implement NIC sessions to the new interface
(dde_ipxe, wifi, usb, nic_bridge, OpenVPN, ...), and thereby unified all
networking components within Genode.
Enhanced tracing facilities
===========================
Recent Genode-based system scenarios like the one described in Section
[Genode as day-to-day operating system] consist of dozens of components that
interact with each other. For reasoning about the behaviour of such scenarios
and identifying effective optimization vectors, tools for gathering a holistic
view of the system are highly desired.
With the introduction of our light-weight
[https://genode.org/documentation/release-notes/13.08#Light-weight_event_tracing - event-tracing facility]
in version 13.08, we laid the foundation for such tools. The current release
extends core's TRACE service with the ability to obtain statistics about CPU
utilization. More specifically, it enables clients of core's TRACE service to
obtain the execution times of trace subjects (i.e., threads). The execution
time is delivered as part of the 'Subject_info' structure. In addition to the
execution time, the structure delivers the information about the affinity of
the subject with a physical CPU.
At the current stage, the feature is available solely on NOVA since this is
our kernel of choice for using Genode as our day-to-day OS. On all other base
platforms, the returned execution times are 0. To give a complete picture of
the system's threads, the kernel's idle threads (one per CPU) are featured as
trace subjects as well. Of course, idle threads cannot be traced but their
corresponding trace subjects allow TRACE clients to obtain the idle time of
each CPU.
By obtaining the trace-subject information in periodic intervals, a TRACE
client is able to gather statistics about the CPU utilization attributed to
the individual threads present (or no longer present) in the system. One
instance of such a tool is the new trace-subject reporter located at
_os/src/app/trace_subject_reporter_. It acts as a TRACE client, which delivers
the gathered trace-subject information in the form of XML-formatted data to a
report session. This information, in turn, can be consumed by a separate
component that analyses the data. In contrast to the low-complexity
trace-subject reporter, which requires access to the privileged TRACE services
of core, the (potentially complex) analysing component does not require access
to core's TRACE service. So it isn't as critical as the trace-subject monitor.
The first representative of a consumer of trace-subject reports is the
CPU-load display mentioned in Section [CPU-load monitoring] and depicted in
Figure [nano3d].
In addition to the CPU-monitoring additions, the tracing facilities received
minor refinements. Up to now, it was not possible to trace threads that use a
CPU session other than the component's initial one. A specific example is
VirtualBox, which employs several CPU sessions, one for each priority. This
problem has been solved by associating the event logger of each thread with
its actual CPU session. Consequently, the tracing mechanism has become able to
trace VirtualBox, which is pivotal for our further optimizations.
Low-complexity software rendering functions
===========================================
Our ambition to use Genode as our day-to-day OS raises the need for custom
graphical applications. Granted, it is principally possible to base such
applications on Qt5, which is readily available to native Genode components.
However, for certain applications like status displays, we prefer to avoid the
dependency on an overly complex GUI tool kit. To accommodate such
applications, Genode hosts a small collection of low-complexity graphics
functions called painters. All of Genode's low-complexity graphical components
such as nitpicker, launchpad, window decorator, or the terminal are based on
this infrastructure.
With the current release, we extend the collection with two new painters
located at _gems/include/polygon_gfx_. Both draw convex polygons with an
arbitrary number of points. The shaded-polygon painter interpolates the color
and alpha values whereas the textured-polygon painter applies a texture to the
polygon. The painters are accompanied by simplistic 3D routines located at
_gems/include/nano3d/_ and a corresponding example (_gems/run/nano3d.run_).
[image nano3d]
With the nano3d demo and our new CPU load display, the screenshot above shows
two applications that make use of the new graphics operations.
Device drivers
##############
Completing the transition to the new platform driver
====================================================
Until now, the platform driver on x86-based machines was formed by the ACPI
and PCI drivers. The ACPI driver originally executed the PCI driver as a slave
(child) service. The ACPI driver parsed the ACPI tables and provided the
relevant information as configuration during the PCI-driver startup. We
changed this close coupling to the more modern and commonly used
[https://genode.org/documentation/release-notes/14.02#New_session_interface_for_status_reporting - report_rom mechanism].
When the new ACPI driver finishes the ACPI table parsing, it provides the
information via a report to any interested and registered components. The
report contains among other the IRQ re-routing information. The PCI driver is
a component, which - according to its session routing configuration - plays
the role of a consumer of the ACPI report.
With this change of interaction of ACPI and PCI driver, the policy for devices
must be configured solely at the PCI driver and not at the ACPI driver. The
syntax, however, stayed the same as introduced with release 15.05.
Finally, the PCI driver 'pci_drv' got renamed to 'platform_drv' as already
used on most ARM platforms. All files and session interfaces containing
PCI/pci in the names were renamed to Platform/platform. The x86 platform
interfaces moved to _repos/os/include/platform/x86/_ and the implementation of
the platform driver to _repos/os/src/drivers/platform/x86/_.
An example x86 platform configuration snippet looks like this:
!<start name="acpi_drv" >
! <resource .../>
! <route>
! ...
! <service name="Report"> <child name="acpi_report_rom"/> </service>
! </route>
!</start>
!
!<start name="acpi_report_rom" >
! <binary name="report_rom"/>
! <resource .../>
! <provides> <service name="ROM" /> <service name="Report" /> </provides>
! <config>
! <rom> <policy label="platform_drv -> acpi" report="acpi_drv -> acpi"/> </rom>
! </config>
! <route> ... </route>
!</start>
!
!<start name="platform_drv" >
! <resource name="RAM" quantum="3M" constrain_phys="yes"/>
! <provides> <service name="Platform"/> </provides>
! <route>
! <service name="ROM">
! <if-arg key="label" value="acpi"/> <child name="acpi_report_rom"/>
! </service>
! ...
! </route>
! <config>
! <policy label="ps2_drv"> <device name="PS2"/> </policy>
! <policy label="nic_drv"> <pci class="ETHERNET"/> </policy>
! <policy label="fb_drv"> <pci class="VGA"/> </policy>
! <policy label="wifi_drv"> <pci class="WIFI"/> </policy>
! <policy label="usb_drv"> <pci class="USB"/> </policy>
! <policy label="ahci_drv"> <pci class="AHCI"/> </policy>
! <policy label="audio_drv"> <pci class="AUDIO"/> <pci class="HDAUDIO"/> </policy>
! </config>
!</start>
In order to unify and simplify the writing of run scripts, we added the
commonly used platform configuration to the file
_repos/base/run/platform_drv.inc_. This file may be included by any test run
script in order to setup a default platform driver configuration.
In addition, the snippet provides the following functions:
'append_platform_drv_build_components', 'append_platform_drv_config' and
'append_platform_drv_boot_modules'. The functions add necessary information to
the 'build_components', 'config' and 'boot_modules' run variables. The
_platform_drv.inc_ also contains the distinction between various ARM/x86
platforms and includes the necessary pieces. Hence, run scripts are largely
relieved from platform-specific peculiarities.
The body of an example run script looks like this:
! set build_components { ... }
!
! source ${genode_dir}/repos/base/run/platform_drv.inc
! append_platform_drv_build_components
!
! build $build_components
!
! create_boot_directory
!
! set config { ... }
!
! append_platform_drv_config
!
! append config { ... }
!
! install_config $config
!
! append_platform_drv_boot_modules
!
! build_boot_image $boot_modules
!
! run_genode_until ...
BCM57cxx network cards
======================
During Hack'n Hike 2015, we had access to a server that featured a Broadcom
network card. Therefore Guido Witmond performed the first steps to enable
Broadcom's BCM 57cxx cards. With this preliminary work in place, we were
quickly able to perform the additional steps required to add BCM 57cxx support
to Genode.
VESA driver refinements
=======================
The VESA driver now reports the frame buffer's line width instead of the
visible width to the client. This fixes a possible distortion if these widths
differ, at the cost that content in the right-most area might be invisible in
such cases.
VirtualBox
##########
Policy-based mouse pointer
==========================
In the previous release, we implemented support for the transparent
integration of the guest mouse pointer with nitpicker via the VirtualBox guest
additions and the vbox_pointer component, which is capable of rendering
guest-provided mouse-pointer shapes. Now, we extended vbox_pointer by a
policy-based configuration that allows the selection of ROMs containing the
actual mouse shape based on the nitpicker session label or domain. With this
feature in place, it is possible to integrate several VirtualBox instances as
well as dedicated pointer shapes for specific components. To see the improved
vbox_pointer in action give _run/vbox_pointer_ a shot.
Dynamic adaptation to screen size changes
=========================================
VirtualBox now notifies the guest operating system about screen-size changes
(for example if the user resizes a window, which shows the guest frame
buffer). The VirtualBox guest additions can use this information to adapt the
guest frame buffer to the new size.
SMP support
===========
Guest operating systems can now use multiple virtual CPUs, which are mapped to
multiple host CPUs. The number of virtual CPUs can be configured in the
'.vbox' file.
Preliminary audio support
=========================
At some point, the use of VirtualBox as a stop-gap solution for using Genode
as everyday OS raises the need to handle audio. With this release, we address
this matter by enabling preliminary audio support in our VirtualBox port. A
back end that uses the audio-out and audio-in sessions to playback and record
sound samples has been added. It disguises itself as the OSS back end that is
already used by vanilla VirtualBox. Since Genode pretends to be FreeBSD in the
eyes of VirtualBox (because Genode's libc is based on FreeBSD's libc), the
provisioning of an implementation of the OSS back end as used on FreeBSD host
systems is the most natural approach. The audio support is complemented by
adding the necessary device models for the virtual HDA as well as the AC97
devices to our VirtualBox port.
For now, it is vital to have the guest OS configure the virtual device in a
way that considers the current implementation. For example, we cannot
guarantee distortion-free playback or recording if the guest OS uses a period
that is too short, typically 10ms or less. There are also remaining issues
with the mixing/filtering code in VirtualBox. Therefore, we bypass it to
achieve better audio quality. As a consequence, the device model of the VM has
to use the same sample rate as is used by the audio-out and audio-in sessions
(44.1kHz).
Enabling audio support is done be adding
! <AudioAdapter controller="HDA" driver="OSS" enabled="true"/>
to the .vbox file manually or configuring the VM accordingly by using the GUI.
Platforms
#########
Execution on bare hardware (base-hw)
====================================
Bender chain loader on base-hw x86_64
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
On Intel platforms, we use the Bender chain loader from the
[https://github.com/alex-ab/morbo - Morbo multiboot suite] to detect available
COM ports of PCI plug-in cards, the AMT SOL device, or as fall back the
default comport 1. The loader stores the I/O port information of the detected
cards into the BIOS data area (BDA), from where it is retrieved by core on
boot and subsequently used for logging. With this release, we added the BDA
parsing to base-hw on x86-64 and enabled the feature in the run tool. As a
prerequisite, we had to fix an issue in bender triggered by the loading of
only one (large) multi-boot kernel. Consequently, its binary in
_tool/boot/bender_ was updated.
Revised page-table handling
~~~~~~~~~~~~~~~~~~~~~~~~~~~
One of the main advantages of the base-hw platform is that the memory trading
concept of Genode is universally applied even with regard to kernel objects.
For instance, whenever a component wants to create a thread, it pays for the
thread's stack, UTCB, and for the corresponding kernel object. The same
applies to objects needed to manage the virtual address space of a component
with the single exception of page tables.
Normally, when the quota, which was donated by a component to a specific
service, runs out, the component receives an exception the next time it tries
to invoke the service. The component can respond by upgrading the respective
session quota. However, in the context of page-fault resolution, this is
particularly difficult to do. The allocation and thereby the shortage of
memory becomes evident only when the client produces a page fault. Therefore,
there is no way to inform the component to upgrade its session quota before
resolving the fault.
Instead of designing a sophisticated protocol between core and the other
components to solve this problem, we decided to simplify the current
page-fault resolution by using a static set of page-tables per component.
Formerly, page tables were dynamically allocated from core's memory allocator.
Now, an array of page tables gets allocated during construction of a
protection domain. When a component runs out of page tables, all of its
mappings get flushed, and the page tables are populated from scratch. This
change greatly simplifies the page-table handling inside of base-hw.
Dynamic interrupt mode setting on x86_64
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
On x86-based hardware, user-level device drivers have become able to specify
the trigger mode and polarity of the interrupts when requesting an IRQ
session. On ARM, those session parameters are ignored. This change enables the
x86_64 platform to support devices, which use arbitrary trigger modes and
polarity settings, e.g., AHCI on QEMU and real hardware.
Fiasco.OC
=========
Genode's device-driver support when using the Fiasco.OC kernel as base
platform received an upgrade.
First, principle support for the Raspberry Pi was added. To make this platform
useful in practice, a working USB driver is important. I.e., the network
interface is connected via USB. Hence the USB driver got enabled for
Fiasco.OC, too. As a result, Genode's software stack can now be used on the
Raspberry Pi by using either our custom base-hw kernel or Fiasco.OC.
Second, support for the Odroid-X2 platform using the Exynos4412 SoC was added,
which includes the drivers for clock management (CMU), power management
(PMU) as well as USB.
Thanks to Reinier Millo Sánchez and Alexy Gallardo Segura for having
contributed this line of work.
Removal of deprecated features
##############################
We dropped the support for the *ARM Versatile Express* board from the Genode
source tree to relieve our automated testing infrastructure from supporting a
platform that remained unused for more than two years.
The device driver environment kit (DDE Kit) was originally intended as a
common API among the execution environments of ported user-level device
drivers. However, over the course of the past years, we found that this
approach could not fulfill its promise while introducing a number of new
problems. We reported our experiences in the release notes of versions
[https://genode.org/documentation/release-notes/12.05#Re-approaching_the_Linux_device-driver_environment - 12.05] and
[https://genode.org/documentation/release-notes/14.11#Roundup - 14.11].
To be able to remove the DDE-Kit API, we reworked the USB driver, our port of
the Linux TCP/IP stack, and the wireless driver accordingly.

1429
doc/release_notes/15-11.txt Normal file
View File

File diff suppressed because it is too large Load Diff

652
doc/release_notes/16-02.txt Normal file
View File

@@ -0,0 +1,652 @@
===============================================
Release notes for the Genode OS Framework 16.02
===============================================
Genode Labs
With version 16.02, we add RISC-V to Genode's supported CPU architectures,
enable the secure pass-through of individual USB devices to virtual machines,
and update the support for the Muen and seL4 kernels.
Trustworthy hardware becomes an increasingly pressing problem. With each new
generation of today's commodity hardware comes a dramatic increase of
complexity, the addition of proprietary companion processors, and opaque
firmware blobs. Even with a perfectly secure operating system, the user's
privacy and security remains at risk as there is no way to assess the
trustworthiness of our underlying hardware. RISC-V is a new hardware
architecture that tries to overcome this problem by the means of open source
and transparency. It is designed to scale from micro controllers to
general-purpose computers, and to be both synthesizable as FPGA softcores and
implementable in ASICs. The prospect of a scalable and trustworthy open-source
hardware platform motivated us to add RISC-V to Genode's supported CPU
architectures. Section [New support for the RISC-V CPU architecture] gives a
brief overview of this line of work.
Thanks to the growing number of our regular developers using Genode as day to
day OS, we create a natural incentive to address typical desktop-OS work
flows. In particular, the new version comes with the ability to assign
individual USB devices to VirtualBox instances. Conceptually, this looks like
a relatively straight-forward feature. But as discussed in Section
[Assignment of USB devices to virtual machines], we had to overcome a number of
challenging problems caused by the inherently dynamic nature of USB-device
hot-plugging. Also on the account of day-to-day computing, the GUI stack
received welcomed usability improvements like keyboard shortcuts for certain
window-management operations.
With respect to Genode's underlying base platforms, we are happy to announce
the updates of the Muen and seL4 kernels. The Muen separation kernel received
an update to version 0.7, which accommodates Genode's regular work flows (via
run scripts) much better than the previous version. As described in Section
[Muen separation kernel], this change clears the way to subject Muen to
Genode's regular automated tests. The seL4 kernel represents an exciting
playground as a future base platform for Genode. We have updated the kernel to
version 2.1, which prompted us to fundamentally revisit the low-level resource
management of Genode on this kernel. A summary of this undertaking is presented
in Section [seL4 version 2.1].
According to the [https://genode.org/about/road-map - road map], we originally planned to
revise the framework API in this release. Even though this topic is
[https://github.com/genodelabs/genode/issues/1832 - very actively pursued], we
decided to not rush it. We find it important to provide a smooth migration path
from the old API to the new one. Determining the best path is actually trickier
than revising the API, though. To let our decisions settle a bit, we postpone
the transition to the upcoming release.
Assignment of USB devices to virtual machines
#############################################
As a migration strategy for running Genode on a daily basis, using VirtualBox
to execute a feature-rich OS is vital. In release
[https://genode.org/documentation/release-notes/15.05#USB-device_pass-through_support - 15.05],
we added USB pass-through support to VirtualBox by enabling its integrated USB
proxy service. Since we use the open-source edition of VirtualBox, we were
merely able to use the OHCI device model and were therefore limited to using
USB 1.x devices in low and full speed mode only. To make matters worse, when
using the OHCI controller model, it is difficult if not impossible to access
USB mass-storage devices. Usually, VirtualBox facilitates the EHCI or xHCI
device models for the pass-through of storage devices. Unfortunately, those
models are only available as a proprietary extension, which cannot be used by
our VirtualBox port.
Having support for the pass-through of high-speed and super-speed USB devices
is a must in such controller models. Therefore, we either have to implement
these models ourselves or port existing ones from another VMM or emulator to
fill the gap. We went for porting existing models first because device-model
development from scratch could end up being time consuming if we want to
guarantee them to work with a variety of different OS drivers.
QEMU xHCI device model
----------------------
QEMU features a NEC xHCI (UPD720200) device model that works well with Windows
guests. For this reason, we decided to give porting this device model a shot.
We applied the DDE approach and started by creating a QEMU emulation
environment so that only the bare minimum amount of source code needed to be
taken from the QEMU sources. It came down to a handful of source files, mainly
the USB core and the xHCI device model files. We iteratively extended the
emulation environment until the QEMU sources compiled and linked fine. One
particular cumbersome issue we had to overcome was the emulation of the QEMU
Object Model. Since QEMU is written in C, it uses its own object model to
implement inheritance. This object model is used throughout QEMU. We took the
easy way out and just used a C++ wrapper class that contains all QEMU objects
that are used in the USB subsystem.
The next step was to develop a USB host device model. This model connects a
USB device attached to Genode's USB host-controller driver to the xHCI device
model. Lucky for us, QEMU already contains a USB host device model that uses
libusb, which we could use as blueprint. We implemented a USB host device that
leverages Genode's custom USB session interface. This host device reacts to a
USB device report coming from another component such as the host-controller
driver. It tries to claim all devices it finds in that report and then creates
a QEMU USB device for each of them that is attached to the xHCI device model.
The xHCI device model needs infrastructure that normally is provided by QEMU
itself such as a timer queue and PCI device handling. We introduced a QEMU
USB controller interface _repos/libports/include/qemu/usb.h_ whose back-end
library interface has to be implemented by a component, i.e. the VMM, that
wants to use the library.
In the end, this work resulted in a small library that contains the xHCI
device model and works in a standalone way. All required resources have to be
provided by the component using the library. This makes it easy to integrate
the library in different VMMs because the user of the library is not forced to
employ the library in a certain way but free to use it any way he chooses.
xHCI device model wrapper in VirtualBox
---------------------------------------
We implemented an xHCI device model _repos/port/src/virtualbox/devxhci.cc_ in
VirtualBox that merely wraps the QEMU USB library and provides the back-end
functionality required by the library to glue QEMU's xHCI device model to
VirtualBox. For now, this device is always part of a VM because there is
currently no way to disable it from within the VirtualBox configuration
front end. Therefore, it is necessary to always give VirtualBox access to a
_usb_devices_ ROM module.
We removed the afore mentioned USB proxy service from our VirtualBox port
because it became redundant with the advent of our xHCI device model.
USB device report filter
------------------------
With the xHCI support in VirtualBox in place, we had to come up with a
mechanism to select, which USB devices it may access. Since USB devices are
usually hot-plugged by the user of the system, we need to be able to configure
the access permissions dynamically at run-time. On this account, we created a
component that intercepts the report from the USB host-controller driver. On
the one hand, this USB device report-filter component screens the device
report coming from the USB host-controller driver by checking each reported
device against a given white list of devices. Only approved devices are
reported to a consumer of the report, i.e. VirtualBox. On the other hand, this
component generates a new configuration for the USB host-controller driver.
The configuration has to be changed each time the filter component finds a
suitable device because the driver will hand out access to a given device to a
client only if there is a valid policy. As we do not know in advance, which
devices might be plugged in, this policy must be maintained dynamically. The
report filter will send the device report only if the host-controller driver
has changed its configuration. This ensures that a matching policy will be in
effect at the time when the client component tries to access the device.
The configuration of the report-filter component can also be changed at run
time.
See _repos/os/src/app/usb_report_filter/README_ for more details on how the
USB device report filter may be configured.
Example configuration
---------------------
The following figure illustrates the interplay and configuration of the
involved components:
[image qemu_xhci]
When the user plugs in a USB device, the USB host-controller driver generates
a device report that is consumed by the USB device report-filter component
(1). The filter component then examines the report and checks if it contains a
device it should report to its report consumer. It then reconfigures the
host-controller driver (2). Afterwards it sends a report to its consumer (3).
The consumer, in this case a VMM, then accesses the USB device (4).
New support for the RISC-V CPU architecture
###########################################
We became aware of [https://riscv.org - RISC-V] when attending several talks
about the project at [https://fosdem.org - FOSDEM] in 2015. RISC-V aims to be
an open-source hardware architecture and is now complemented by many projects
that target the release of real hardware or ASICs (for example,
[https://www.lowrisc.org - the LowRISC project]). We have experience with various
major CPU architectures and many systems on a chip and, therefore, embrace a
sharp eye on certain platform properties. Intel's ME and ARM's Trustzone
practically lock out operating systems of certain hardware and firmware
features. The true nature of these mechanisms becomes increasingly dubious,
especially when trying to build a secure open-source operating system. Intel's
AMT technology for instance comes with a complete TCP/IP stack that intercepts
packets from the integrated NIC and a VNC server that can magically expose a
mouse and a keyboard at the USB controller. If you are interested in more
details about this topic
[https://blog.invisiblethings.org/papers/2015/x86_harmful.pdf - Intel x86 considered harmful]
by Joanna Rutkowska is a very good read. We decided to have a deeper look at
the RISC-V architecture as an alternative open hardware platform. Especially,
since the LowRISC project promises a completely open system on chip, including
the peripherals.
RISC-V comes with a lot of optional features, so it can cover a large field of
applications reaching from simple I/O processors to general-purpose computing.
For example, there are 64 and 32 bit ISA (instruction set architecture)
versions, three page table formats with the option to omit paging at all, up
to four privilege modes, and a minimal integer core ISA (I). Everything else,
like multiplication and division (M), atomic instructions (A), and floating
point support (F) are subject to ISA extensions and are completely optional
for a specific hardware implementation.
For Genode, we chose to add the RISC-V support to our custom _base-hw_ kernel.
Since Genode may be used as a general purpose OS, we implemented the kernel
using the 64 bit RISC-V version, the Sv39 three-level page table format, and
the so-called general-purpose extension (G), which is the abbreviation for the
IAMF extensions. The current implementation provides the kernel and the
necessary adaptations of the user level part of core.
For testing, we used the RISC-V instruction emulator called
[https://github.com/riscv/riscv-isa-sim - Spike]. There also exists a RISC-V
implementation for various Zynq FPGAs. Genode's Zynq board support has kindly
been added and contributed by Mark Vels.
In the current state, basic Genode applications including core, init, and
components that use shared libraries can be executed on top of our RISC-V
port. We did not enable the libc and postponed further activity as the
platform currently does not specify the interaction with peripherals.
Steps to test Genode on RISC-V
------------------------------
# Building the instruction emulator
! # download the front end server
! git clone https://github.com/ssumpf/riscv-fesvr.git
!
! # build the front end server
! cd riscv-fesvr
! mkdir build
! cd build
! export RISCV=<installation path>
! ../configure --prefix=$RISCV
! (sudo) make install
!
! # download the instruction emulator
! cd ../../
! git clone https://github.com/ssumpf/riscv-isa-sim.git
! cd riscv-isa-sim
!
! # build the emulator
! mkdir build
! cd build
! ../configure --prefix=$RISCV --with-fesvr=$RISCV
! (sudo) make install
!
! # add $RISCV/bin to path
! export PATH=$RISCV/bin:$PATH
# Building Genode and running a test scenario
! # download Genode
! cd ../../
! git clone https://github.com/genodelabs/genode.git
!
! # build the Genode tool chain
! cd genode
! ./tool/tool_chain riscv
!
! # create RISC-V build directory
! ./tool/create_builddir hw_riscv
! cd build/hw_riscv
!
! # build and execute the printf run script
! make run/printf
GUI stack usability improvements
################################
Motivated by the daily use of Genode as desktop OS by an increasingly number
of developers, the window-layouter component of the
[https://genode.org/documentation/release-notes/15.11#GUI_stack - GUI stack]
received welcomed usability improvements.
Configurable window placement
-----------------------------
The policy of the window layouter can be adjusted via its configuration. For
a given window label, the window's initial position and its maximized state
can be defined as follows:
! <config>
! <policy label="mupdf" maximized="yes"/>
! <policy label="nit_fb" xpos="50" ypos="50"/>
! </config>
Keyboard shortcuts
------------------
The window layouter has become able to respond to key sequences. However,
normally, the layouter is not a regular nitpicker client but receives only
those input events that refer to the window decorations. It never owns the
keyboard focus. In order to propagate global key sequences to the layouter,
nitpicker must be explicitly configured to direct key sequences initiated with
certain keys to the decorator. For example, the following nitpicker
configuration routes key sequences starting with the left windows key to the
decorator. The window manager, in turn, forwards those events to the layouter.
! <start name="nitpicker">
! ...
! <config>
! ...
! <global-key name="KEY_LEFTMETA" label="wm -> decorator" />
! ...
! </config>
! ...
! </start>
The response of the window layouter to key sequences can be expressed in the
layouter configuration as follows:
! <config>
! <press key="KEY_LEFTMETA">
! <press key="KEY_TAB" action="next_window">
! <release key="KEY_TAB">
! <release key="KEY_LEFTMETA" action="raise_window"/>
! </release>
! </press>
! <press key="KEY_LEFTSHIFT">
! <press key="KEY_TAB" action="prev_window">
! <release key="KEY_TAB">
! <release key="KEY_LEFTMETA" action="raise_window"/>
! </release>
! </press>
! </press>
! <press key="KEY_ENTER" action="toggle_fullscreen"/>
! </press>
! </config>
Each '<press>' node defines the policy when the specified 'key' is pressed.
It can be equipped with an 'action' attribute that triggers a window action.
The supported window actions are:
:next_window: Focus the next window in the focus history.
:prev_window: Focus the previous window in the focus history.
:raise_window: Bring the focused window to the front.
:toggle_fullscreen: Maximize/unmaximize the focused window.
By nesting '<press>' nodes, actions can be tied to key sequences. In the
example above, the 'next_window' action is executed only if TAB is pressed
while the left windows-key is kept pressed. Furthermore, key sequences can
contain specific release events. In the example above, the release of the left
windows key brings the focused window to front, but only if TAB was pressed
before.
Device drivers
##############
USB host-controller driver enhancements
=======================================
The _usb_drv_ component now solely uses a policy to grant other components
access to USB devices exposed by its raw interface (USB session). On the basis
of the 'label' attribute, it will choose a pre-configured device that is
identified by either the 'bus' and 'dev' or the 'vendor' and 'product'
attribute tuple. To accommodate policy decisions made at run time, the USB
driver is now able to reload its configuration on demand. The USB device
report now contains a 'bus' and a 'dev' attribute as well in order to identify
a USB device more precisely. In addition to that, there is also a generated
'label' attribute in form of 'usb-<bus>-<dev>' that may be used to form
policies while configuring the system dynamically, e.g., when using the
_usb_report_filter_ component.
USB mass-storage driver
=======================
Up to now, access to USB storage devices was provided by the USB
host-controller driver only. However, its ability to do so is limited. E.g.,
it only supports one storage device and the storage device cannot be changed
at run-time. With this release we add a USB mass-storage driver that supports
UMS bulk-only devices that use the SCSI Block Commands set (direct-access).
This is still most common for USB sticks. Devices using different command
sets, e.g SD/HC devices or some external disc drives, will not work properly
if at all. The driver uses the USB session interface to access the USB device
and provides its service as block session to its client.
This component is part of the first step providing the ability to mount and
use USB sticks dynamically when using Genode as a general purpose OS. In the
future, the _usb_drv_ component should solely be the host-controller driver
while other tasks are handled by dedicated USB driver components such as this
one.
Audio output on Linux
=====================
The audio-out driver for Linux was modernized by replacing its multi-threaded
architecture by an event-driven architecture using Genode's server API. In
addition, the playback is now driven by a timer. For now it is a periodic
timer that triggers every 11 ms which is roughly the current audio-out period.
The driver now also behaves like the other BSD-based audio-out driver, i.e.,
it always advances the play pointer. That is vital for the audio-out stack
above the driver to work properly (e.g., the mixer).
Libraries and applications
##########################
New Genode-world repository
===========================
With a growing number of users and contributors comes the desire to bring more
and more existing software to Genode. Most of such libraries and applications,
however, are outside of the scope of Genode as an OS framework. In contrast to
device drivers, protocol stacks, and low-level OS services, which we subject
to our regular automated tests, most 3rd-party software is pretty independent
from Genode. The attempt to integrate the growing pool of such diverse
software into the main repository does not scale.
For this reason, we introduce the new
[https://github.com/genodelabs/genode-world - Genode World] repository, which
is the designated place for hosting ported applications, libraries, and games.
To use it, you first need to obtain a clone of Genode:
! git clone https://github.com/genodelabs/genode.git genode
Now, clone the _genode-world.git_ repository to _genode/repos/world:_
! git clone https://github.com/genodelabs/genode-world.git genode/repos/world
By placing the _world_ repository under the _repos/_ directory, Genode's tools
will automatically incorporate the ports provided by the _world_ repository.
For building software of the _world_ repository, the build-directory
configuration _etc/build.conf_ must be extended with the following line:
! REPOSITORIES += $(GENODE_DIR)/repos/world
*Word of caution*
In contrast to the components found in the mainline Genode repository, the
components within the _world_ repository are not subjected to the regular
quality-assurance measures of Genode Labs. Hence, problems are to be expected.
If you encounter bugs, build problems, or stability issues, please report them
to the [https://github.com/genodelabs/genode-world/issues - issue tracker] or
the [https://genode.org/community/mailing-lists - mailing list].
Updated 3rd-party software
==========================
The following 3rd-party code packages of the _ports_ and _libports_
repositories have been ported or updated:
* Lynx 2.8.8rel.2 (noux package)
* OpenSSH 7.1p1 (noux package)
* tar-1.27 (noux package)
* libssh 0.7.2
* Lighttpd 1.4.38
Platforms
#########
Execution on bare hardware (base-hw)
====================================
Within the last months, the initialization code of our custom kernel got
re-arranged to simplify the addition of new architectures, e.g., the RISC-V
port (Section [New support for the RISC-V CPU architecture]) while also making
its implementation leaner. A positive side effect of this work was the
generalization of multi-processor and L2-cache support for ARM's Cortex-A9
CPUs. For instance, the Wandboard (Freescale i.MX6 SoC) is now driven with all
four cores, and its memory can be accessed with full speed.
Besides those feature additions, we fixed an extremely rare and tricky race
condition in the implementation of the kernel-protected capabilities,
introduced in release 15.05. A capability's lifetime within a component is
tracked by a reference-counting like mechanism that is under control of the
component itself. When the kernel transfered a capability to a component, and
the very same capability was deleted within the component simultaneously, the
received capability was marked as invalid, which led to diverse, sporadic
faults. This deficit in the capabilities reference-counting is solved with the
current release.
Muen separation kernel
======================
Build integration
-----------------
Building Genode scenarios running on top of the
[https://muen.sk - Muen separation kernel] has been greatly simplified by
properly integrating the Muen system build process into the Genode build system.
As described in the
[https://genode.org/documentation/release-notes/15.08#Genode_on_top_of_the_Muen_Separation_Kernel - 15.08 release notes],
the architecture with Muen is different since the entire hw_x86_64_muen Genode
system runs as a guest VM on top of the separation kernel. This means that the
Genode base-hw image must itself be packaged into the final Muen system image
as an additional step after the Genode system build.
The packaging process of a Muen system image is performed by the new
_image/muen_ run-tool plugin, which processes the following RUN_OPT parameters.
:--image-muen-external-build:
Muen system is built automatically or externally
:--image-muen-system:
Muen system policy
:--image-muen-components:
Muen system components required for the given system policy
:--image-muen-hardware:
Muen target hardware platform
:--image-muen-gnat-path:
Path to GNAT toolchain
:--image-muen-spark-path:
Path to SPARK toolchain
The options are automatically added to the _etc/build.conf_ file for the
hw_x86_64_muen base-hw platform. The
[https://genode.org/documentation/platforms/muen - documentation] has been
updated to reflect the new, simplified build process.
A port file was added to facilitate the download of the Muen sources v0.7 and
to check the required dependencies.
Using the new _image/muen_ script in combination with iPXE allows to run the
Genode test suite via the autopilot tool.
MSI support
-----------
Muen employs Intel VT-d interrupt remapping (IR) besides DMA remapping for
secure device assignment. As a consequence, PCI devices using Message Signaled
Interrupts (MSI) must be programmed to trigger requests in remappable format
(see Intel VT-d specification, Section 5.1.2.2 for further details).
To enable the use of MSIs with the base-hw kernel, a platform-specific
function has been introduced that returns the necessary MSI parameters for a
given PCI device. If either the platform or the specific device does not
support MSI, the function returns false.
On hw_x86_64_muen, the function consults the Muen subject info page to supply
the appropriate information to the IRQ session. This allows Genode device
drivers to transparently use MSIs for passed-through PCI devices.
seL4 version 2.1
================
By the end of 2015, the [https://sel4.systems/ - seL4 kernel] version 2.0 was
published. With the current release, we update Genode's preliminary support
for this kernel from the experimental branch of one year ago to the master
branch of version 2.1. Note that this line of work is still considered as an
exploration. As of now, there is still a way to go until we can leverage seL4
as a fully featured base platform. Under the hood of Genode, the transition to
the version 2.1 master branch had the following implications.
In contrast to the experimental branch, the seL4 master branch has no way to
manually define the allocation of kernel objects within untyped memory ranges.
Instead, the kernel maintains a built-in allocation policy. This policy rules
out the deallocation of once-used parts of untyped memory. The only way to
reuse memory is to revoke the entire untyped memory range. Consequently, we
cannot share a large untyped memory range for kernel objects of different
protection domains. In order to reuse memory at a reasonably fine granularity,
we need to split the initial untyped memory ranges into small chunks that can
be individually revoked. Those chunks are called "untyped pages". An untyped
page is a 4 KiB untyped memory region.
The bootstrapping of core has to employ a two-stage allocation approach now.
For creating the initial kernel objects for core, which remain static during
the entire lifetime of the system, kernel objects are created directly out of
the initial untyped memory regions as reported by the kernel. The so-called
"initial untyped pool" keeps track of the consumption of those untyped memory
ranges by mimicking the kernel's internal allocation policy. Kernel objects
created this way can be of any size. For example the CNode, which is used to
store page-frame capabilities is 16 MiB in size. Also, core's CSpace uses a
relatively large CNode.
After the initial setup phase, all remaining untyped memory is turned into
untyped pages. From this point on, newly created kernel objects cannot exceed
4 KiB in size because one kernel object cannot span multiple untyped memory
regions. The capability selectors for untyped pages are organized similarly to
those of page-frame capabilities. There is a new 2nd-level CNode
(UNTYPED_CORE_CNODE) that is dimensioned according to the maximum amount of
physical memory (1M entries, each entry representing 4 KiB). The CNode is
organized such that an index into the CNode directly corresponds to the
physical frame number of the underlying memory. This way, we can easily
determine an untyped page selector for any physical addresses, i.e., for
revoking the kernel objects allocated at a specific physical page. The
downside is the need for another 16 MiB chunk of meta data. Also, we need to
keep in mind that this approach won't scale to 64-bit systems. We will
eventually need to replace the PHYS_CORE_CNODE and UNTYPED_CORE_CNODE by CNode
hierarchies to model a sparsely populated CNode. The following figure
illustrates the layout of core's capability space.
[image sel4_core_cspace_master]
Organization of core's capability space on seL4
For each protection domain, core maintains a so-called VM CSpace that holds
capability selectors for page frames and page tables. The size constraint of
kernel objects has the immediate implication that the VM CSpaces of protection
domains must be organized via several levels of CNodes. I.e., as the top-level
CNode of core has a size of 2^12, the remaining 20 PD-specific CSpace address
bits are organized as a 2nd-level 2^4 padding CNode, a 3rd-level 2^8 CNode,
and several 4th-level 2^8 leaf CNodes. The latter contain the actual selectors
for the page tables and page-table entries of the respective PD.
As another slight difference from the experimental branch, the master branch
requires the explicit assignment of page directories to an ASID pool.
Functionality-wise the update to version 2.1 brings no changes. The
preliminary support is still limited to Genode's most fundamental mechanisms
like the bootstrapping, the creation of protection domains, the execution of
threads, and inter-component communication. User-level device drivers are not
supported yet. Such functional improvements are scheduled for Genode 16.08.
Linux
=====
We started to experience crashes of our dynamic linker (ldso) when using
Genode's _base-linux_ platform on recent Linux kernels. Ldso is primarily a
shared object, which is linked to dynamic binaries. But ldso is also an
executable, which, once started loads the dynamically-linked binary along with
all shared libraries required by the binary. Up to now, ldso had to be loaded
at a link address defined at compilation time, which we enforced through
linker-script magic. Unfortunately, this does not work any longer on recent
Linux versions. The kernel notices that ldso is a shared object and loads it
at an arbitrary (randomized) address, which ultimately results in a
segmentation fault during ldso initialization. We found a fix for this issue
by marking ldso as an executable in the ELF header. But since ldso is linked
to all dynamic binaries (it contains Genode's base libraries) the GNU linker
then refused to link because ldso was not marked as a shared object.
Therefore, we decided to implement true self relocation within ldso. This
feature only works on Genode's base-linux platform as it requires some
symbol-address magic.

1315
doc/release_notes/16-05.txt Normal file
View File

File diff suppressed because it is too large Load Diff

1126
doc/release_notes/16-08.txt Normal file
View File

File diff suppressed because it is too large Load Diff

729
doc/release_notes/16-11.txt Normal file
View File

@@ -0,0 +1,729 @@
===============================================
Release notes for the Genode OS Framework 16.11
===============================================
Genode Labs
In contrast to most parts of the framework, the fundamental low-level
protocols, which define the interaction between parent and child components
have remained unchanged since the very first Genode version. From this
interplay, the entire architecture follows. That said, certain initial design
choices were not perfect. They partially resulted from limitations of the
kernels we used during Genode's early years and from our pre-occupation with a
certain style of programming. Over the years, the drawbacks inherent in our
original design became more and more clear and we drafted rough plans to
overcome them. However, reworking the fundamental protocols of a system that
already accommodates hundreds of component implementations cannot be taken
lightly. Because of this discomfort, we repeatedly deferred the topic -
until now. With the rapidly growing workloads carried by Genode, we
deliberately decided to address long-standing deficiencies rather than adding
the features we originally planned according to the
[https://genode.org/about/road-map - road map].
Section [Asynchronous parent-child interactions] presents the reworking of
Genode's component interplay at the lowest level. With this change in place,
we feel much more comfortable to scale up our workloads in the upcoming
releases.
Functionality-wise, the most prominent topic of the current release is the
vastly improved NIC-routing component. Since we introduced the first version
of the NIC router in the previous release, we took an iterative approach to
shape the component according to its most prominent use cases. Section
[Further improved virtual networking] summarizes the changes and the
motivation behind them.
Even though we added support for seL4 in the previous release, the NOVA
hypervisor is still our go-to kernel for x86-based hardware because of its
feature set. For this reason, we continuously improve this kernel and the
NOVA-specific components like VirtualBox. Section [NOVA hypervisor] covers
the introduction of an asynchronous map operation to NOVA.
Further topics of the current release range from added smart-card support,
over a new timeout API, to a VFS-based time-based password generator. With
respect to the road map, we postponed most topics originally planned. In
particular, we intended to enable the use of Genode on top of Xen by following
the footsteps of the existing Muen support - using our custom
base-hw kernel within a Xen DomU domain. However, before proceeding this
route, we decided to modernize the kernel design, in particular with respect
to bootstrapping and address-space management. Some parts of this line of work
are already present in the current release, for example the unification of the
boot-module handling as explained in Section
[Unified handling of boot modules].
Asynchronous parent-child interactions
######################################
When Genode was born in 2006, the L4 microkernels of the time universally
lacked an asynchronous inter-process-communication (IPC) mechanism.
Consequently, we designed the first version of Genode with the presumption
that components had to interact solely synchronously. To us, this seemed to be
the "right" way because the synchronous low-footprint IPC was presumably the
key for L4's good performance. It felt natural to leverage this benefit to the
maximum extent possible.
To illustrate the implications of this line of thinking for Genode, let's take
a look at a simple scenario where a parent component hosts two children and one
child provides a service to the other child.
[image simple_scenario]
During the creation of a session, the kernel's IPC mechanism serves three
purposes. First, it is used to communicate information between different
protection domains, in this case the parent, the client, and the server.
Second, it implicitly dictates the flow of control between the involved
parties because the caller blocks until the callee replies to the IPC call.
Third, the IPC is the mechanism to delegate authority (like the authority to
access the server's session object) between protection domains. The latter is
realized with the kernel's ability to carry capabilities as IPC message
payload. If this sounds a bit too abstract, please consider reviewing Section
3.1. "Capability-based security" of the
[https://genode.org/documentation/genode-foundations-16-05.pdf - Genode Foundations].
Using solely a synchronous IPC mechanism, the sequence of establishing a
session in the given scenario is as follows. In the context of Genode,
we usually refer to synchronous IPC as RPC (remote procedure call).
[image sync_session_seq]
The sequence looks straightforward:
# The client issues an RPC call to its parent, requesting a session for a
service of the given type while also passing a number of session-construction
arguments along with the request.
# Given the service name as provided with the session request, the parent
determines the server to ask for a new session. It requests a session
on behalf of the client by performing an RPC call to the server's prior
registered "root" capability. This capability refers to an interface for
creating and closing sessions.
# The server responds to the invocation of its root interface by creating
a new session object along with a session capability.
Whereas the session object is local to the server, the corresponding
session capability can be passed (delegated) to other components.
Each component in possession of the session capability is able to interact
with the server's corresponding session object via RPC calls.
The server returns the session capability to the parent as the result of the
parent's RPC call.
# The parent forwards the session capability to the client as the result of
the client's original RPC call.
Even though the simplicity of this protocol seems nice, it has inherent
limitations:
First, as the parent performs a synchronous RPC call to the server on behalf
of the client, it must trust the server to eventually respond to the RPC call.
If the server doesn't, the parent may block forever. In contrast to the client
that actually uses the service and thereby relies on the liveliness of the
server, the parent should not need to trust the server to be responsive. To
deal with the risk of an unresponsive server, Genode's existing runtime
environments (like the init component), maintain a dedicated thread for each
child. The session requests originating from a child are handled by the
corresponding parent-local child thread. In the worst case - if the server
fails to respond - only a single child thread stays blocked but the other
parts of the runtime environment remain unaffected. Consequently, runtime
environments have to be multi-threaded components. This, in turn, comes at the
cost of added complexity, in particular the need for error-prone inter-thread
synchronization.
Second, the approach keeps the parent's state implicitly stored in the stacks
of the parent's threads. This becomes a problem in dynamic runtime
environments that need to kill subsystems at arbitrary times. E.g., imagine
the situation where the client component is to be destroyed while the parent's
call to the server's root interface is still pending. The safe destruction of
the child - including its associated parent-local child thread - requires the
parent to abort the RPC call, which is a complex and - again - error-prone
operation.
Third, even though not inherent to synchronous RPC, Genode's original design
facilitated the use of a session capability as argument for requesting the
parent to close a specific session. However, the use of capabilities as
re-identifiable tokens is not well supported by most kernels, including seL4
([https://sel4.systems/pipermail/devel/2014-November/000114.html - discussion]
on the seL4 mailing list).
Asynchronous communication throughout Genode
--------------------------------------------
In 2008, we acknowledged the sole reliance on synchronous RPC as too limiting
and introduced an
[https://genode.org/documentation/release-notes/8.11#Asynchronous_notifications - API for asynchronous notifications].
On the traditional L4 kernels, we implemented the API by using Genode's
core component as a proxy for signal delivery. The use of asynchronous
notifications soon became natural and wide-spread throughout Genode. Today,
most session interfaces combine three forms of inter-component communication,
namely synchronous RPC calls, asynchronous notifications, and shared memory.
The new Genode API introduced in
[https://genode.org/documentation/release-notes/16.05#The_great_API_renovation - version 16.05]
further cultivated the modeling of Genode components as single-threaded state
machines instead of multi-threaded programs.
Still, until now, the most fundamental mechanism of Genode - the protocol
between parent and child components - has remained synchronous. The reasons
are twofold. First, our workaround for realizing runtime environments in a
multi-threaded way worked too well. So we were not constantly bothered by this
design problem. Second and more importantly, redesigning the fundamental
mechanism of the framework while not breaking the more than 300 existing
components is quite scary. But in anticipation of the rapidly scaling
workloads imposed on Genode, we had to take on the problem sooner or later.
We figured that now - with the modernized framework API in place - it's the
right time. From redesigning the interplay of parent and child components, we
will become able to create single-threaded runtime environments that behave
completely deterministically while consuming less resources than
multi-threaded programs. By the explicit enumeration of possible states, we
greatly ease the validation/evaluation of such crucial components.
New session-creation procedure
------------------------------
Following the asynchronous approach, the sequence of creating a session now
looks as follows:
[image async_session_seq]
The dotted lines are asynchronous notifications, which have fire-and-forget
semantics. A component that triggers a signal does not block.
The following points are worth noting:
* Sessions are identified via IDs, which are plain numbers as opposed to
capabilities. The IDs as seen by the client and server belong to different
ID name spaces.
IDs of sessions requested by the client are allocated by the client. IDs
of sessions requested at the server are allocated by the parent.
* The parent does no longer need to perform RPC calls to any of its children.
Hence, the need for multiple threads in runtime environments disappears.
* Each activation of the parent merely applies a state change of the session's
meta data structures maintained at the parent, which capture the entire
state of session requests. There is no hidden state stored on the parent's
stack.
* The information about pending session requests is communicated from the
parent to the server via a ROM session. At startup, the server requests
a ROM session for the ROM module "session_requests" from its parent. The
parent implements this ROM session locally. Since ROM sessions support
versions, the parent can post version updates of the "session_requests"
ROM with the regular mechanisms already present in Genode.
* The involved parties can potentially run in parallel.
Outcome and current state
-------------------------
Intuitively, the sequence of steps required to establish a session has
become more complicated. However, for the users of the framework, the entire
procedure is completely transparent. With a few tricks, we were actually able
to implement this fundamental change while keeping almost all existing
components untouched. One trick is the introduction of a server-local proxy
mechanism, which translates the requests obtained from the "session_requests"
ROM to component-local RPC calls on the server's root interface. So from the
perspective of an existing server component, a session request still looks
like a synchronous RPC request from the outside. Of course, the proxy is meant
as an intermediate solution until we have crafted a convenient front-end API
for the asynchronous mode of operation.
Even though the biggest share of components remains unaffected by the change,
this is not true for all components. In particular, runtime environments had
to be reworked, in some cases quite fundamentally. These include core, init,
noux, the loader, GDB monitor, launcher, CLI monitor, and the platform driver.
The change does not only affect the interplay between components but also
required a reconsideration of the child-creation procedure.
Besides the architectural improvement, this line of work had two welcome
effects.
First, in contrast to the original design, which relied on capabilities as
re-identifiable tokens, the new version greatly alleviates the need for
re-identifying capabilities on seL4. So we are able to eliminate a
long-standing problem with Genode on this kernel.
Second, the work called for new data structures for the safe interaction with
ID spaces (_base/id_space.h_) and object registries (_base/registry.h_). Those
data structures will possibly be useful in a lot of places that currently use
plain (and fairly unsafe) AVL trees or lists.
At the API level, the change is almost transparent to regular components,
except for two details. The upgrading of session quota is no longer
possible by a mere RPC call to the parent. Instead, 'Connection' objects
received a new 'upgrade_ram' method that must be used instead. Speaking
of 'Connection' objects, we had to remove the (fairly obscure) 'KEEP_OPEN'
feature, which is conceptually incompatible with the new design.
Further improved virtual networking
###################################
The
[https://genode.org/documentation/release-notes/16.08#Virtual_networking_and_support_for_TOR - previous release]
introduced the NIC router - a component that individually routes IP
packets between multiple NIC sessions, translates between different IP
subnets, and also supports port forwarding and NAT. For the first version of
the NIC router, we focused on the technical realization. Now, besides
some optimization and restructuring, we took the chance to polish the
configuration interface of the component. The goal was to make the interface
more intuitive and reduce pitfalls to a minimum. Roughly speaking, the
handling of the NIC router became more tailored to its/our typical use cases.
Let's create a practical setup to explain the changes in detail. Assume that
there are two virtual subnets 192.168.1.0/24 and 192.168.2.0/24 within our
Genode system. They connect as Virtnet A and B to the router. The standard
gateway of the virtual networks is the NIC router with IP 192.168.*.1 . The
router's uplink, on the other hand, is connected to the NIC driver. It
interfaces the machine with our real-world home network 10.0.2.0/24. The home
network is connected to the internet through its standard gateway 10.0.2.1.
[image nic_router_basic]
The basic router configuration for this setup without any routing rules would
be as follows:
! <policy label_prefix="virtnet_a" domain="virtnet_a" />
! <policy label_prefix="virtnet_b" domain="virtnet_b" />
!
! <domain name="uplink" interface="10.0.2.55/24" gateway="10.0.2.1" />
! <domain name="virtnet_a" interface="192.168.1.1/24" />
! <domain name="virtnet_b" interface="192.168.2.1/24" />
The first thing to notice is the changed usage of the policy tag. Previously,
the policy label - normally solely designated to correlate sessions with
configuration domains - was misused also as unique peer identifier in the
routing rules. This approach disregarded advanced label-matching techniques
such as the 'label_prefix' used above. Now, the whole NIC-router-specific
enhancement of the policy tag moved to the new '<domain>' tag, leaving the
policy tag only with its original purpose to select policies. Note that even
if this modification gives the impression, the router is not yet capable of
handling multiple NIC sessions at one domain at a time.
In the domain tag, the 'interface' attribute replaces the old policy attribute
named 'src'. That means, it tells the router which IP identity to use when
talking as itself to the domain. But in addition to that, the 'interface'
attribute also defines which subnet this identity and the domain belong to.
This reflects a basic decision we made during the reworking process: The new
NIC router is aware of subnets. Sessions of the same subnet have the same
configuration domain. We came to this conclusion as it solves some fundamental
problems with the old version. First, the equivalence of domain and subnet
enables us to link a default gateway to a subnet by adding the 'gateway'
attribute to the domain tag. In our example, this is done in the uplink
domain. The 'gateway' attribute is optional for a domain and replaces the
former 'via' attributes of the different routing rules. It is more efficient
and natural to have this value set only once at the corresponding subnet than
having it scattered all over the routing rules of the remote domains as done
before. If a domain has no default gateway, it drops all packets with a
foreign recipient.
The second advantage of a domain being equivalent to a subnet is that handling
ARP broadcasts becomes easy. It can be excluded that such ARP broadcasts
concern sessions outside the source domain anymore. And as sessions in the
same domain are not distinguishable to the routing, the broadcast can be sent
to all of them without breaking any rules.
Now, let's enhance our example by some routing rules. One pretty complicated
thing to do with the old NIC router was port forwarding. You had to combine
different routing rules, explicitly enable the back routing at the remote
side, and take care that NAT was applied - a lot of opportunities for
mistakes. With the new version, it became easier. Let's assume we have an HTTP
server in Virtnet A and an NTP server in Virtnet B. We want the NIC router to
act as proxy for their services in our home network.
[image nic_router_servers]
In order to achieve this, the uplink domain must be enhanced by two rules:
! <policy label_prefix="virtnet_a" domain="virtnet_a" />
! <policy label_prefix="virtnet_b" domain="virtnet_b" />
!
! <domain name="uplink" interface="10.0.2.55/24" gateway="10.0.2.1" />
! <tcp-forward port="443" domain="virtnet_a" to="192.168.1.2" />
! <udp-forward port="123" domain="virtnet_b" to="192.168.2.2" />
! </domain>
!
! <domain name="virtnet_a" interface="192.168.1.1/24" />
! <domain name="virtnet_b" interface="192.168.2.1/24" />
The TCP forwarding rule for port 443 (HTTP+TLS/SSL) redirects to IP address
192.168.1.2 in Virtnet A and the UDP forwarding rule for port 123 (NTP)
redirects to IP address 192.168.2.2 in Virtnet B. The Virtnet domains remain
empty as the router keeps track of the redirected transfers and routes back
reply packets automatically. Also automatically, the router applies NAT for the
server as it is in the nature of port forwarding.
Next, we add some clients to Virtnet B that like to talk to our home network
and the internet. We want them to be hidden via NAT when they do so. For
internet communication, they shall furthermore be limited to HTTP+TLS/SSL and
IMAP+TLS/SSL.
[image nic_router_client]
This is what the router configuration looks now:
! <policy label_prefix="virtnet_a" domain="virtnet_a" />
! <policy label_prefix="virtnet_b" domain="virtnet_b" />
!
! <domain name="uplink" interface="10.0.2.55/24" gateway="10.0.2.1" />
! <tcp-forward port="443" domain="virtnet_a" to="192.168.1.2" />
! <udp-forward port="123" domain="virtnet_b" to="192.168.2.2" />
! <nat domain="virtnet_b" tcp-ports="1000" udp-ports="1000">
! </domain>
!
! <domain name="virtnet_a" interface="192.168.1.1/24" />
! <domain name="virtnet_b" interface="192.168.2.1/24" >
! <tcp dst="10.0.2.0/24"> <permit-any domain="uplink" /> </tcp>
! <udp dst="10.0.2.0/24"> <permit-any domain="uplink" /> </udp>
! <tcp dst="0.0.0.0/0">
! <permit port="443" domain="uplink" />
! <permit port="993" domain="uplink" />
! </tcp>
! </domain>
There are several new tag types. One of them is the NAT configuration for
Virtnet B in the uplink domain. In contrast to the former NIC-router version
where NAT settings were part of the source domain, NAT is now configured in
the target domain with a sub-tag for each source. This has the advantage
of supporting heterogeneous NAT configurations for a packet source depending
on which domain it talks to. Besides, it is more intuitive to read. Apart from
that, the NAT settings haven't changed.
Furthermore, there are the new TCP and UDP tags in the Virtnet-B domain. The
first two of them have a 'permit-any' sub-tag. With this combination, we open
all ports to IP addresses of the 10.0.2.0/24 subnet, our home network, and
route them to the uplink domain. TCP packets that don't match these first two
rules may fall back to the third. This TCP rule doesn't have all ports opened
but only 443 (HTTP+TLS/SSL) and 993 (IMAP+TLS/SSL). Both ports are again bound
to the uplink domain. As the IP filter 0.0.0.0/0 of the surrounding rule isn't
restrictive, we now also route packets to a foreign destination. The NIC
router redirects such packets to the default gateway of our home network.
Compared to the old router version where IP and UDP/TCP routing had to be
combined for this purpose, the new TCP and UDP rules with their
port-permission sub-rules have some notable advantages. Like port-forwarding
rules, TCP and UDP rules always imply link-state tracking in order to route
back reply packets automatically. This can be seen also in our example as no
further routing rules had to be added to the uplink domain. This aspect is
clear from the outermost rule and not dependent on sub-rules anymore.
Furthermore, the strict separation of UDP and TCP routing prevents
configuration faults and increases readability. Last but not least, the
'permit-any' rule allows something new. Opening all ports for an address range
was previously only possible without link-state tracking as it could be
expressed only on the IP level.
At this point, we have thoroughly discussed the layer-3 routing abilities of
the new NIC router and our focus has indeed moved more into this direction.
Even though IP routing is still available, we found that it should be more
clearly separated from the rest. To illustrate this feature, we enhance our
example again. We want the Virtnets to be allowed to communicate to each other
without any restrictions. For that purpose, we add two more rules to the
router configuration:
! <policy label_prefix="virtnet_a" domain="virtnet_a" />
! <policy label_prefix="virtnet_b" domain="virtnet_b" />
!
! <domain name="uplink" interface="10.0.2.55/24" gateway="10.0.2.1" />
! <tcp-forward port="443" domain="virtnet_a" to="192.168.1.2" />
! <udp-forward port="123" domain="virtnet_b" to="192.168.2.2" />
! <nat domain="virtnet_b" tcp-ports="1000" udp-ports="1000">
! </domain>
!
! <domain name="virtnet_a" interface="192.168.1.1/24" />
! <ip dst="192.168.2.0/24" domain="virtnet_b"/>
! </domain>
!
! <domain name="virtnet_b" interface="192.168.2.1/24" >
! <tcp dst="10.0.2.0/24"> <permit-any domain="uplink" /> </tcp>
! <udp dst="10.0.2.0/24"> <permit-any domain="uplink" /> </udp>
! <tcp dst="0.0.0.0/0">
! <permit port="443" domain="uplink" />
! <permit port="993" domain="uplink" />
! </tcp>
! <ip dst="192.168.1.0/24" domain="virtnet_a"/>
! </domain>
As you can see, each of the new IP rules in the Virtnet domains match the
addresses of the opposite subnet and route to the corresponding domain. As
mentioned, the new IP rules and UDP/TCP rules are not combined anymore to
clearly distinguish IP routing from layer-3 routing because this decision has
far-reaching effects. First, in contrast to UDP and TCP routing, IP routing is
stateless. Thus, for each IP routing rule one has to be sure to have a
back-routing rule at the remote domain or else bidirectional communication
won't happen. And second, NAT does not apply to IP-routed packets. So, if
you're not aware of such packets, you may unintentionally reveal information
about a private network.
For more details on the new NIC router, you may refer to the comprehensive
documentation in the _repos/os/src/server/nic_router/README_ file and the
basic NIC-router test at _libports/run/nic_router.run_ .
Base framework
##############
Improved RPC mechanism
======================
Since we introduced Genode's current API for synchronous RPCs in
[https://genode.org/documentation/release-notes/11.05#New_API_for_type-safe_inter-process_communication - version 11.05],
inter-component communication within Genode has become almost a child's play.
The RPC framework leverages the C++ type system and templates to a great
effect. In contrast to the traditional use of IDL compilers, the interaction
with RPC objects provided by other components is robust and natural because
no language boundaries need to be crossed.
Still, a few differences between RPC calls and regular function calls remain.
In particular, there exist a few restrictions with regard to the types of
RPC function arguments. Those types did not just need to be POD (plain old
data) types but they had to be default-constructible, too. Whereas the former
restriction still applies (non-POD objects that include references or
vtables cannot be used as arguments), the latter limitation has been lifted
now. Generally, non-default-constructible types are a way to attain
simpler code because the special case of an "invalid" object does not need
to be considered. I.e., values of such types can be kept as constants as
opposed to variables. If an object exists (as equivalent to successful
instantiation), it is valid. With the improved RPC mechanism, the RPC
framework does no longer stay in the way in this respect.
Thanks to Edgard Schmidt for this welcome contribution!
Unification and tightening of session labels
============================================
In Genode, each session requested by a client component is labeled according
to the components that intermediate the session request. The client can
optionally specify a label of choice along with the session request. Its
parent prefixes the client-provided label by a label of its own. If the
session request is further passed to the parent's parent, the grandparent
prepends its own label. This works recursively. Consequently, the final label
as seen by the server is the product of the labeling policies of all
components on the route of the session request.
The label is used for two purposes. First, the server uses the label as
a key for a server-side policy selection. E.g., depending on the session label
received by the disk-partition server, the server decides which partition to
hand out to the client. Second, the label is used by intermediate components
to take session-routing decisions. E.g., based on the label of a file-system
session request, a parent component may route the request to one of several
file-system servers.
Originally, Genode did not impose a specific way of how labels are formed.
It was up to each intermediate component to filter the label of a session
request in any way desired. However, in practice, this freedom remained unused
and the very simple successive prefixing of labels prevails in all our use
cases. Each intermediate node concatenates its own label in front of the label
supplied by the originator of the session request. The different parts of the
label are separated with the character sequence '" -> "'. Some corner cases
were handles specially for aesthetic reasons. For example, if a client
provided no label, the parent would skip the pending separator. That said,
since each intermediate component had to provide the labeling policy, not all
components were consistent in these respects. Since we found no use for
arbitrary labeling policies, we decided to make the only prominent way of
session labeling mandatory for all intermediate components. We thereby removed
the aesthetically motivated corner cases and possible ambiguities. I.e., with
the original policy, it was not possible to distinguish a unlabeled session
requested by a client from a labeled session requested by the client's parent.
As a consequence, the stricter labeling must now be considered wherever
a precise label was specified as a key for a session route or a server-side
policy selection. The simplest way to adapt those cases is to use a
'label_prefix' instead of the 'label' attribute. Alternatively, the
'label' attribute may used by appending '" -> "' (note the whitespace).
Transition to new framework API
===============================
Since we fundamentally revised Genode's API in
[https://genode.org/documentation/release-notes/16.05#The_great_API_renovation - version 16.05],
we gradually adapt our existing components. Given that Genode comes with
over 300 components, this is no small feat. But with 30 percent of the
components converted, we already made substantial progress.
In some respects, the conversion is actually nearly complete. In particular,
the move away from format-string-based text output to our new type-safe output
facility has been applied to almost all components now. The former 'PDBG'
macro that is quite useful for temporary debug messages has been replaced with
a new version that must be manually included via the _base/debug.h_ header
file. Like the regular log functions, the new PDBG facility uses the type-safe
text-output facility.
Minor API adjustments
---------------------
While applying Genode's new API, we refined the API in the following respects:
We added a dedicated 'String' constructor overload to better accommodate
string literals. This overload covers the common case for initializing a
string from a literal without employing the 'Output' mechanism. This way, such
strings can by constructed without calling virtual functions, which in turn
makes the 'String' usable during the self-relocation phase of the dynamic
linker.
Up till now, several Genode components still rely on the use of 'snprintf'
whenever strings must be assembled out of smaller pieces. As we like to shun
format strings from Genode altogether, we needed an alternative mechanism.
Since we introduced the new type-safe text-output facilities in Genode 16.05,
there is an obvious solution: Let the 'String' constructor accept an arbitrary
list of arguments, which are turned into their respective textual
representation and appear concatenated in the resulting string. Consequently,
strings can be assembled with the same flexibility as log output. For the
construction of 'String' objects from character buffers of a known size, the
'Cstring' utility can be used, which takes a 'char const *' and an optional
length as arguments.
Several low-level types received support for the new output facilities, e.g.,
'Xml_node' or the network-related headers in _os/net/_.
In anticipation of the forthcoming package-management infrastructure, we try
to unify Genode's executable binaries across kernels and architectures
wherever reasonable. Of course, the latter is not possible with respect to the
used instructions. But unifying symbol information is deemed worthwhile. For
this reason, we changed the 'Genode::size_t' type to be always defined as an
'unsigned' 'long'. This is in contrast to GCC's built-in '__SIZE_TYPE__',
which is defined as 'unsigned int' on 32-bit architectures but 'unsigned long'
on 64-bit architectures.
OS-level infrastructure and device drivers
##########################################
New timeout-handing API
=======================
The new timeout API offers tools for easily multiplexing a single time
source among different timeouts. In general, the time source can be
implemented individually but we expect that the most prominent use case will
be the multiplexing of timer sessions. Thus, the timeout library also provides
a convenience tool for this use case. A library-usage example can be found
under _os/src/test/timeout_. If you're interested in implementing
your own time source, you can find an example at _os/include/os/timer.h_ .
Support for smart cards
=======================
We ported the [https://pcsclite.alioth.debian.org/pcsclite.html - PC/SC Lite]
library to Genode, which provides a commonly used API for communicating with
smart cards. It supports USB smart card readers, using the
[https://pcsclite.alioth.debian.org/ccid.html - CCID] library as driver.
The CCID driver itself requires [https://libusb.info - libusb] to access the
USB device.
Vanilla PC/SC Lite is structured as a client-server architecture, consisting
of the 'pcscd' daemon, which runs on a privileged user account and manages all
card reader devices, and one or more non-privileged client applications, which
communicate with pcscd to access the card readers. On Genode, pcscd's role as
privileged device manager is not really needed, since the devices can also be
managed using Genode's configuration mechanisms. For this reason, we merged
the part of pcscd which implements the API with the pcsc-lite client library.
In the current state, a Genode application using PC/SC Lite can access a single
card reader device, which is selected using its USB product ID and vendor ID in
the application's configuration and in the policy of the USB driver.
More configuration details can be found in the README files of the PC/SC Lite,
CCID, and libusb libraries in the libports repository and in the accompanying
_smartcard.run_ script.
Libraries and applications
##########################
Time-based password generation
==============================
A time-based one-time password authentication client that adheres to the
Google Authenticator standard has been introduced into the
[https://github.com/genodelabs/genode-world - world repository].
Single use, time-based passwords are commonly used as an additional
authentication step for web-based services. In this scheme, a user generates
and presents a six digit passcode to a service generated using a shared secret
and a timestamp. This short passcode length makes manual entry convenient so
that the shared secret may be stored on a separate device than the service
client, such as a smartphone, layering the security properties of both
devices.
The 'gtotp' VFS plugin provides these passcodes by embedding the generator as
a special file in the file-system layer of a component. This approach provides
readily available passcodes for programmatic and manual use without enlarging
the code base to encompass a GUI, command-line, or networked interface.
At the time of this release, the common use case is to manually retrieve codes
for clients running in VirtualBox by reading special files with an isolated
instance of the Noux runtime. Storing the shared secret on the same device
contradicts the recommendations of the standard but the trade-off is that the
software stack required to host the shared secret is significantly smaller
than that found on a mobile device.
Random number generator testing
===============================
No random number generator can be proved to be good, but empirical statistical
tests can prove that some are bad. A port of the TestU01 RNG test suite is
provided in the world repository. The TestU01 batteries give independent
assurance of the fitness of Genode's CPU jitter based RNG and are available
for testing future physical and non-phyical RNGs.
VirtualBox on top on the NOVA hypervisor
########################################
Both VirtualBox-based virtual machine monitors on Genode got updated to the
latest revision as provided by Oracle, namely 4.3.40 and 5.1.10 - mainly to
stay close to the upstream versions.
Platforms
#########
Unified handling of boot modules
================================
Until now, the way of passing boot modules from the boot procedure to the core
component, which core provides as ROM modules, varied from platform to
platform. Either we used a multiboot-compliant bootloader that accepts
multiple modules, or the platform provided some specific way of linking binary
modules together with the kernel, e.g., the Elfweaver tool of OKL4.
By unifying the boot-module handover, we further reduce platform specific core
code. Thereby, maintenance costs are decreased, and code analysis becomes
easier. With this new solution, when issuing to build the core component:
! make core
within the build system, only a core library gets built. Not until all
binaries needed by a run-script are available, a final image is linked
together using the core library and all additional binaries. The core
component now can access its ROM modules directly via addresses contained in
its binary. As a side effect of this change, there is no core binary in the
'bin' or 'core' directory of the corresponding build directory available
anymore. Instead, you will find the core binary with no ROM modules, but
including debug information under 'var/run/*.core' within your build
directory. The concrete name depends on the name of the run-script.
The new approach is used on all platforms except Linux where the ROM modules
still need to be accessed via the file-system.
NOVA hypervisor
===============
We extended the kernel to support the asynchronous delegation of kernel
resources. Up to now, resources could only be delegated during RPC or during
the initial protection-domain construction. With this extension, the
construction and setup of new protection domains, threads, and especially
virtual CPUs for the VirtualBox VMM became more straightforward and several
quirks inside the 'core' component could be dropped. The added kernel syscall
expects the NOVA-kernel capabilities of the source and target protection
domains, which effectively renders the operation solely available to 'core' -
as only holder of the NOVA protection domain capabilities.
Additionally, we changed the CPU ID enumeration in Genode/NOVA to a
predictable order. The lower CPU IDs used via the Genode 'Cpu_session'
interface now correspond to the first hyper-thread of all physical CPU cores.
For example, on a quad-core machine with hyper-threading enabled Genode's CPU
IDs 0-3 refer to the first hyper-threads of all physical cores and IDs 4-7 to
the second hyper-threads.

0
doc/release_notes-17-02.txt → doc/release_notes/17-02.txt
View File

1033
doc/release_notes/17-05.txt Normal file
View File

File diff suppressed because it is too large Load Diff

651
doc/release_notes/17-08.txt Normal file
View File

@@ -0,0 +1,651 @@
===============================================
Release notes for the Genode OS Framework 17.08
===============================================
Genode Labs
The flagship feature of Genode 17.08 has been in the works for more than a
year: The support for hardware-accelerated graphics on Intel Gen-8 GPUs. This
is an especially challenging topic because it is riddled with terminology,
involves highly complex software stacks, carries a twisted history with it,
and remains to be a moving target. It took up a lot of patience to build up a
profound understanding of the existing driver architectures and the mechanisms
offered by modern graphics hardware. On the other hand, with the proliferation
of hardware-based sandboxing features like virtual GPU memory and hardware
contexts, we found that now is the perfect time for a clean-slate design of a
microkernelized GPU driver.
Section [Hardware-accelerated graphics for Intel Gen-8 GPUs] introduces this
work, which includes our new GPU multiplexer as well as the integration with
the client-side Mesa protocol stack.
The second focus of the current release is the extension of Genode's supported
base platforms. Most prominently, we upgrade the seL4 kernel to version 6.0
while extending the architecture support from 32-bit x86 to ARM and 64-bit
x86 (Section [The seL4 kernel on ARM and 64-bit x86 hardware]). To bring
Genode closer to cloud-computing scenarios, we added basic support for
executing Genode scenarios as Xen DomU domains (Section [Genode as Xen DomU]).
Furthermore, the Muen separation kernel has been updated to a current version.
As a cross-kernel effort, there is work under way to boot Genode-based
systems via UEFI, currently addressing the NOVA, base-hw, and seL4 kernels.
Among the many other functional additions are a new VFS plugin for accessing
FAT file systems, new components like _sequence_ and _fs_report_ that aid new
system compositions, and our evolving custom package-management
infrastructure.
Hardware-accelerated graphics for Intel Gen-8 GPUs
##################################################
The ability to leverage hardware-accelerated graphics is generally taken for
granted in modern commodity operating systems. The user experience of
modern desktop environments, web-browser performance, and obviously games
depend on it. On the other hand, the benefit of hardware-accelerated graphics
comes at the expense of tremendous added complexity in the lower software
stack, in particular in system components that need to be ultimately trusted.
For example, with circa 100 thousand lines of code, the Intel GPU driver in
the Linux kernel is an order of magnitude more complex than a complete modern
microkernel. In a monolithic-kernel-based system, this complexity is
generally neglected because the kernel is complex anyway. But in
microkernel-based scenarios optimized for a trusted computing base in the
order of a few ten thousand lines of code, it becomes unacceptable.
Fortunately, recent generations of graphics hardware provide a number of
hardware features that promise to solve this conflict, which prompted us to
investigate the use of these features for Genode.
During this year's Hack'n'Hike event, we ported the ioquake3 engine to Genode.
As preliminary requirement, we had to resurrect OpenGL support in our aging
graphics stack and enable support for current Intel HD Graphics devices (IGD).
We started by updating Mesa from the old 7.8.x to a more recent 11.2.2 release.
Since we focused mainly on supporting Intel devices, we dropped support for the
Gallium back end as Intel still uses the old DRI infrastructure. This decision,
however, also influenced the choice of the software rendering back end. Rather
than retaining the softpipe implementation, we now use swrast. In addition, we
changed the available OpenGL implementation from OpenGL ES 2.x to the fully
fledged OpenGL 4.5 profile, including the corresponding shader language
version. As with the previous Mesa port, EGL serves as front end API for
system integration and loads a DRI back-end driver (i965 or swrast). EGL
always requests the back-end driver 'egl_drv.lib.so' in form of a shared
object. Genode's relabeling features are used to select the proper back end
via a route configuration. The following snippet illustrates such a
configuration for software rendering:
! <start name="gears" caps="200">
! <resource name="RAM" quantum="32M"/>"
! <route>
! <service name="ROM" label="egl_drv.lib.so">
! <parent label="egl_swrast.lib.so"/>
! </service>
! <any-service> <parent/> <any-child/> </any-service>
! </route>
! </start>
With the graphics-stack front end in place, it was time to take care of the
GPU driver. In our case this meant implementing the DRM interface in our
ported version of the Intel i915 DRM driver. Up to now, this driver was solely
used for mode setting while we completely omitted supporting the render
engine.
[image mesa_genode]
With this new and adapted software stack, we successfully could play ioquake3
on top of Genode with a reasonable performance in 1080p on a Thinkpad X250.
During this work, we gathered valuable insights into the architecture of a
modern 3D-graphics software stack as well as into recent Intel HD Graphics
hardware. We found that the Intel-specific Mesa driver itself is far more
complex than its kernel counter part. The DRM driver is mainly concerned with
resource and execution management whereas the Mesa driver programs the GPU.
For example, amongst others, Mesa compiles the OpenGL shaders into a
GPU-specific machine code that is passed on to the kernel for execution.
While inspecting the DRM driver, it became obvious that one of the reasons for
its complexity is the need to support a variety of different HD Graphics
generations as well as different features driven by software-usage patterns.
For our security related use cases, it is important to offer a clear isolation
and separation mechanism per client. Hardware features provided by modern
Intel GPUs like per-process graphics translation tables (PPGTT) and hardware
contexts that are unique for each client make it possible to fulfill these
requirements.
By focusing on this particular feature set and thus limiting the supported
hardware generations, the development of a maintainable GPU multiplexer for
Genode became feasible. After all, we strive to keep all Genode components as
low complex as possible, especially resource multiplexers like such a GPU
multiplexer.
[image intel_gpu_drv]
This image shows multiple GPU-session clients and the resources they are
using. The fence registers as well as the aperture is partitioned between
them, the PPGTT is backed by the system memory, and the contexts are located
in disjoint GGTT regions.
Within four months, we implemented an experimental GPU multiplexer for Intel
HD Graphics Gen8 (Broadwell class) devices. We started out defining a GPU
session interface that is sufficient to implement the API used by the DRM
library. For each session, the driver creates a context consisting of a
hardware context, a set of page tables (PPGTT), and a part of the aperture.
The client may use the session to allocate and map memory buffers used by the
GPU. Each buffer is always eagerly mapped 1:1 into the PPGTT by using the
local virtual address of the client. Special memory buffers like an image
buffer are additionally mapped through the aperture to make use of the
hardware-provided de-tiling mechanism. As is essential in Genode components,
the client must donate all resources that the driver might need to fulfill the
request, i.e., quota for memory and capability allocations. Clients may
request the execution of their workload by submitting an execution buffer. The
GPU multiplexer will then enqueue the request and schedule all pending
requests sequentially. Once the request is completed, the client is notified
via a completion signal.
[image multi_gl]
Example scenario of multiple OpenGL programs that use the new GPU multiplexer
for hardware-accelerated rendering.
We consider this first version of the GPU driver as experimental. As of now,
it only manages the render engine of the GPU. Mode-setting or rather display
handling must be performed by another component. Currently, the VESA driver is
used for this purpose. It also lacks any power-management functionality and
permanently keeps the GPU awake. Both limitations will be addressed in future
releases and support for Gen9+ (Skylake) and newer devices might be added.
In its current incarnation, the GPU multiplexer component consists of about
4,200 lines of code whereas the Mesa DRI i965 driver complements the driver at
the client side with about 78,000 lines of code.
The seL4 kernel on ARM and 64-bit x86 hardware
##############################################
With the 16.08 release, we brought the seL4 support to a level to be
considered being on par with the other supported kernels. At the time,
Genode's use of seL4 was limited to 32-bit x86 platforms.
In the current release, we extend the platform support to ARM and 64-bit x86.
We started this line of work with an incremental kernel upgrade from version
3.2.0 to 5.2.0 and finally to seL4 6.0. Through these upgrades, we were able
to drop several Genode-specific seL4 patches, which were required in the 16.08
release. One major improvement of version 6.0 compared to earlier versions is
the handling of device-memory announcements by the kernel to Genode's roottask
_core_.
With the kernel update in place, we inspected the x86-specific part thoroughly
while splitting and separating it properly into architecture-agnostic and
architecture-dependent parts. Upon this work, we added the
architecture-specific counterparts for x86_64 and ARM. One major work item was
to make the page-table handling in Genode's core aware and generic enough to
handle the different page-table sizes of the three architectures.
For the ARM support, we decided to enable the i.MX6 FreeScale based SoC,
namely the Wandboard Quad board. Since the seL4 kernel interface provides no
timeout support, we revived a user-level timer driver that we originally
developed for our custom base-hw kernel: The so-called EPIT timer, which is
part of most i.MX SoCs.
We finished the essential work for the mentioned three platforms in
less time than expected and, thereby, had spare time to address additional
features.
First, we enabled multiprocessor support for Genode/seL4 on x86 and
thread-priority support for all seL4 platforms. Additionally, we were able to
utilize the seL4 benchmark interface for Genode's trace infrastructure in
order to obtain utilization information about threads and CPUs. The Genode
components _top_ (text-based) and _cpu_load_monitor_ (graphical) are now
usable on Genode/seL4.
Finally, as we are currently exploring the support for booting various kernels
via UEFI on x86, we took the chance to investigate the steps needed to boot
seL4 via UEFI. UEFI firmware does not always provide a compatibility support
module (CSM) for legacy BIOS boot support. Hence, we extended the seL4 kernel
for Genode according to the Multiboot2 specification, which enables us to
start Genode/seL4 together with GRUB2 - as an UEFI capable bootloader - on
machines missing CSM support.
Base framework and OS-level infrastructure
##########################################
Simplified IOMMU handling
=========================
When IOMMUs are used on x86, all host memory targeted via direct memory
accesses (DMA) by devices must eagerly be registered in the respective I/O
page table of the device. Up to now, Genode supports IOMMUs on NOVA only. On
this kernel, a device protection domain is represented as a regular protection
domain with its virtual memory layout being used for both the CPU's MMU and
the device. Traditionally, mappings into such virtual memory spaces are
inserted on demand as responses to page faults. However, as there are no page
faults for DMA transactions, DMA buffers must always be eagerly mapped. The
so-called device PD hid this gap for NOVA. In anticipation of adding IOMMU
support for more kernels, we desired to generalize the device-PD mechanism by
introducing an explicit way to trigger the insertion of DMA memory into the
proper page tables.
We extended the PD-session interface by a 'map' function, which takes a
virtual memory region of the PD's virtual address space as argument. The page
frames of the previously attached dataspaces are added eagerly by core to the
IOMMU page-tables. With this explicit 'map' support, we were able to replace
the Genode/NOVA-specific device-PD implementation with a generic one, which
will easily accommodate other kernels in the future.
New report server for capturing reports to files
================================================
The report session is a simple mechanism for components to publish structured
data without the complexity of a file-system layer. In the simplest case, a
client component will produce a report and communicate it directly to a
component acting as a server. The disadvantage is that the report client
becomes reliant on the liveliness and presence of the consumer component. So
in the more robust case, the _report_rom_ component acts as the server hosting
the report service as well as a ROM service for components consuming reports.
The _report_rom_ server permits ROM access only to clients matching an
explicit configuration policy. This is good for security but opaque to a user.
Reports can only be read where an explicit policy is in place and only a
single report session can report to an active ROM session.
The new _fs_report_ component is a friendlier and more flexible report server.
Reports are written to a file system using a file and directory hierarchy that
expresses session routing. This allows for intuitive report inspection and
injection via a file system. When used with the _ram_fs_ and _fs_rom_ servers,
it can also replicate the functionality of _report_rom_.
New runtime environment for starting components sequentially
============================================================
The _init_ component is a prime example of software with an emphasis on
function over features. It is the fundamental building block for combining
components yet its behavior is simple and without heuristics. Like other
contemporary init managers, it starts components in parallel, but to a more
extreme degree in that it has no concept of "runlevels" or "targets", all
components are started as soon as possible. The concrete sequence of execution
is instead determined by when server components make service announcements and
how quickly they respond to client requests.
In some cases, the execution of one component must not occur until the
execution of another component ends, be it that the first produces output that
is consumed by the second, or that the two contend for a service that cannot
be multiplexed. Init contains no provisions to enforce ordering. But we are
free to define new behaviors in other management components.
The solution to the problem of ordering is the _sequence_ component. Sequence
walks a list of children and executes them in order, one at a time. With only
one child active, there is no need for any local resource or routing
management. By applying the same session label transformations as init,
external routing and policy handling are unchanged.
An example of ordering a producer and consumer within an init configuration
follows:
! <start name="sequence">
! <resource name="RAM" quantum="128M"/>
! <config>
! <start name="producer">
! <config .. />
! </start>
! <start name="consumer">
! <config .. />
! </start>
! </config>
! <route>
! <service name="LOG" label_prefix="producer">
! <child name="log_a"/> </service>
! <service name="LOG" label_prefix="consumer">
! <child name="log_b"/> </service>
! <any-service> <parent/> <any-child/> </any-service>
! </route>
! </start>
Support for boot-time initialized frame buffer
==============================================
UEFI-based systems do not carry along legacy BIOS infrastructure, on which
our generic VESA driver depends. Hence, when booting via UEFI, one has to use
either a hardware-specific driver like our Intel-FB driver or - alternatively -
facilitate generic UEFI mechanisms.
Instead of booting in VGA text mode and leaving the switch to a graphics mode
(via real-mode SVGA BIOS subroutines) to the booted OS, UEFI employs the
so-called graphics output protocol as a means to setup a reasonable default
graphics mode prior booting the operating system. In order to produce
graphical output, the operating system merely has to know the physical address
and layout of the frame buffer. Genode's core exposes this information as the
_platform_info_ ROM module. The new _fb_boot_drv_ driver picks up this
information to provide a Genode framebuffer session interface. Hence, on
UEFI-based systems, it can be used as a drop-in replacement for the VESA
driver. In contrast to the VESA driver, however, it is not able to switch
graphics modes at runtime.
The new component is located at _os/src/drivers/framebuffer/boot/_. Thanks
to Johannes Kliemann for this contribution.
Extended non-blocking operation of the VFS
==========================================
In
[https://genode.org/documentation/release-notes/17.02#VFS_support_for_asynchronous_I_O_and_reconfiguration - version 17.02],
we added support for non-blocking reads from the VFS in the form of the
'read_ready()', 'queue_read()', and 'complete_read()' functions. Since then,
it has become obvious that blocking within the VFS is not only problematic in
the VFS server itself when multiple clients are connected, but also when the
VFS is deployed in a multi-threaded environment and a VFS plugin needs to
reliably wait for I/O-completion signals.
For this reason, we reworked the interface of the VFS even more towards
non-blocking operation and adapted the existing users of the VFS accordingly.
The most important changes are:
* Directories are now created and opened with the 'opendir()' function and
the directory entries are read with the 'queue_read()' and 'complete_read()'
functions.
* Symbolic links are now created and opened with the 'openlink()' function and
the link target is read with the 'queue_read()' and 'complete_read()'
functions and written with the 'write()' function.
* The 'write()' function does not wait for signals anymore. This can have the
effect that data written by a VFS library user has not been processed by a
file-system server when the library user asks for the size of the file or
closes it (both done with RPC functions at the file-system server). For this
reason, a user of the VFS library should request synchronization before
calling 'stat()' or 'close()'. To make sure that a file-system server has
processed all write request packets that a client submitted prior the
synchronization request, synchronization is now requested at the file-system
server with a synchronization packet instead of an RPC function. Because of
this change, the synchronization interface of the VFS library has been split
into the 'queue_sync()' and 'complete_sync()' functions.
Making block sessions read-only by default
==========================================
Genode server components are expected to apply the safest and strictest
behavior when exposing cross-component state or persistent data. In practice
block and file-system servers only allow access to clients with explicitly
configured local policies. The file-system servers enforce an additional
provision that sessions are implicitly read-only unless overridden by policy.
This release introduces a similar restriction to the AHCI driver and partition
multiplexer. Clients of these servers require an affirmative 'writeable'
attribute on policies to permit the writing of blocks. Write permission at
these servers may also be revoked by components that forward block-session
requests by placing 'writeable="no"' into session-request arguments.
All users of _ahci_drv_ and _part_blk_ are advised that this change may break
existing configurations without explicit 'writeable' policies.
Refined time handling
=====================
Release 17.05 introduced a
[https://genode.org/documentation/release-notes/17.05#New_API_for_user-level_timing - new API for user-level timing]
named _timeout framework_. Together with this new framework came a
comprehensive test that stresses all aspects of the interface. During the past
few months, this test has turned out to be an enrichment for Genode far beyond
its original scope. As the test significantly raised the standards in
user-level timing, it also sharpened our view on the measurement precision of
various timer drivers and timestamps, which act as input for the framework.
This revealed several problems previously unidentified. For instance, we
improved the accuracy and stability of the time values provided by the drivers
for the Raspberry-Pi timer, the Cortex-A9 timer, the PIT, and the LAPIC. We
also were able to further optimize the calibration of the TSC in the NOVA
kernel.
Additionally, the test also helped us to refine the timeout framework itself.
The initial calibration of the framework - that previously took about 1.5
seconds - is now performed much quicker. This makes microseconds-precise time
available immediately after the timer connection switched to the modern
fine-grained mode of operation, which is a prerequisite for hardware drivers
that need such precision during their early initialization phase. The
calculations inside the framework also became more flexible to better fit the
characteristics of all the hardware and kernels Genode supports.
Finally, we were able to extend the application of the timeout framework. Most
notably, our C runtime uses it as timing source to the benefit of all
libc-using components. Another noteworthy case is the USB driver on the
Raspberry Pi. It previously couldn't rely on the default Genode timing but
required a local hardware timer to reach the precision that the host
controller expected from software. With the timeout framework, this workaround
could be removed from the driver.
FatFS-based VFS plugin
======================
Genode has supported VFAT file-systems since the 9.11 release when the
[http://elm-chan.org/fsw/ff/00index_e.html - FatFS] library was first ported.
The 11.08 release fit the library into the libc plugin architecture and
in 12.08 FatFS was used in the _ffat_fs_ file-system server. Now, the 17.08
release revisits FatFS to mold the library into the newer and more flexible
VFS plugin system. The _vfs_fatfs_ plugin may be fitted into the VFS server or
used directly by arbitrary components linked to the VFS library. As the
collection of VFS plugins in combination with the VFS file-system server has a
lower net maintenance cost than multiple file-system servers, the _ffat_fs_
server will be retired in a future release.
Enhanced GUI primitives
=======================
Even though we consider Qt5 as the go-to solution for creating advanced
graphical user interfaces on top of Genode, we also continue to explore an
alternative approach that facilitates Genode's component architecture to an
extreme degree. The so-called menu-view component takes an XML description of
a dialog as input and produces rendered pixels as output. It also gives
feedback to user input such as the hovered widget at a given pointer position.
The menu view does not implement any application logic but is meant to be
embedded as a child component into the actual application. This approach
relieves the application from the complexity (and potential bugs) of widget
rendering. It also reinforces a rigid separation of a view and its underlying
data model.
The menu view was first introduced in
[https://genode.org/documentation/release-notes/14.11#New_menu_view_application - version 14.11].
The current release improves it in the following ways:
* The new '<float>' widget aligns a child widget within a
larger parent widget by specifying the boolean attributes 'north', 'south',
'east', and 'west'. If none is specified, the child is centered. If opposite
attributes are specified, the child is stretched.
* A new '<depgraph>' widget arranges child widgets in the form of a
dependency graph, which will be the cornerstone for Genode's upcoming
interactive component-composition feature. As a prerequisite for
implementing the depgraph widget, Genode's set of basic graphical primitives
received new operations for drawing sub-pixel-accurate anti-aliased lines
and bezier curves.
* All geometric changes of the widget layout are animated now. This includes
structural changes of the new '<depgraph>' widget.
[image depgraph]
The menu-view component is illustrated by the run script at
_gems/run/menu_view.run_.
C runtime
=========
The growing number of ported applications used on Genode is accompanied by the
requirement of extensive POSIX compatibility of our C runtime. Therefore, we
enhanced our implementation by half a dozen features (e.g., O_ACCMODE
tracking) during the past release cycle. We thank the contributors of patches
and test cases and will continue our efforts to accommodate more ported
open-source components in the future.
Libraries and applications
##########################
Mesa adjustments
================
The Mesa update required the adaption of all components that use OpenGL.
In particular that means the Qt5 framework. Furthermore, we also enabled
OpenGL support in our SDL1 port.
As playground, there are a few OpenGL examples. The demos are located under
_repos/libports/src/test/mesa_demos_, which use the EGLUT bindings. There
are also some SDL based examples in the world repository under
_repos/world/src/test/sdl_opengl_.
Package management
==================
The previous release featured the initial version of Genode's
[https://genode.org/documentation/release-notes/17.05#Package_management - custom package-management tools].
Since then, we continued this line of work in three directions.
First, we refined the depot tools and the integration of the depot with our
custom work-flow ("run") tool. One important refinement is a simplification of
the depot's directory layout for library binaries. We found that the initial
version implied unwelcome complexities down the road. Instead of placing
library binaries in a directory named after their API, they are now placed
directly in the architecture directory along with regular binaries.
Second, driven by the proliferated use of the depot by more and more run
scripts, we enhanced the depot with new depot recipes as needed.
Third, we took the first steps to use the depot on-target. The experimentation
with on-target depots is eased by the new 'create_tar_from_depot_binaries'
function of the run tool, which allows one to assemble a new depot in the form
of a tar archive out of a subset of packages. Furthermore, the new
_depot_query_ component is able to scan an on-target depot for runtime
descriptions and returns all the information needed to start a subsystem based
on the depot content. The concept is exemplified by the new
_gems/run/depot_deploy.run_ script, which executes the "fs_report" test case
supplied via a depot package.
Platforms
#########
Genode as Xen DomU
==================
We want to widen the application scope of Genode by enabling users to easily
deploy Genode scenarios on Xen-based cloud platforms.
As a first step towards this goal, we enhanced our run tool to support running
Genode scenarios as a local Xen DomU, managed from within the Genode build
system on Linux running as Xen Dom0.
The Xen DomU runs in HVM mode (full virtualization) and loads Genode from an
ISO image. Serial log output is printed to the text console and graphical
output is shown in an SDL window.
To use this new target platform, the following run options should be defined in
the 'build/x86_*/etc/build.conf' file:
! RUN_OPT = --include boot_dir/$(KERNEL)
! RUN_OPT += --include image/iso
! RUN_OPT += --include power_on/xen
! RUN_OPT += --include log/xen
! RUN_OPT += --include power_off/xen
The Xen DomU is managed using the 'xl' command line tool and it is possible to
add configuration options in the 'xen_args' variable of a run script. Common
options are:
* Disabling the graphical output:
! append xen_args { sdl="0" }
* Configuring a network device:
! append xen_args { vif=\["model=e1000,mac=02:00:00:00:01:01,bridge=xenbr0"\] }
* Configuring USB input devices:
! append xen_args { usbdevice=\["mouse","keyboard"\] }
Note that the 'xl' tool requires super-user permissions. Interactive
password input can be complicated in combination with 'expect' and is not
practical for automated tests. For this reason, the current implementation
assumes that no password input is needed when running 'sudo xl', which can
be achieved by creating a file '/etc/sudoers.d/xl' with the content
! user ALL=(root) NOPASSWD: /usr/sbin/xl
where 'user' is the Linux user name.
Execution on bare hardware (base-hw)
====================================
UEFI support
------------
Analogously to our work on the seL4 and NOVA kernels in this release, we
extended our base-hw kernel to become a Multiboot2 compliant kernel. When used
together with GRUB2, it can be started on x86 UEFI machines missing legacy
BIOS support (i.e., CSM).
RISC-V
------
With Genode version 17.05, we updated base-hw's RISC-V support to privileged
ISA revision 1.9.1. Unfortunately, this implied that dynamic linking was not
supported on the RISC-V architecture anymore. Since dynamic linking is now
required for almost all Genode applications by default, this became a severe
limitation. Therefore, we revisited our RISC-V implementation - in particular
the kernel entry code - to lift the limitation of being able to execute only
statically linked binaries.
Additionally, we integrated the Berkeley Boot Loader (BBL), which bootstraps
the system and implements the machine mode, more closely into our build
infrastructure. We also added a new timer implementation to base-hw by using
the _set timeout SBI_ call of BBL.
What still remains missing is proper FPU support. While we are building the
Genode tool chain with soft float support, we still encounter occasions where
FPU code is generated, which in turn triggers compile time errors. We will
have to investigate this behavior more thoroughly, but ultimately we want to
add FPU support for RISC-V to our kernel and enable hardware floating point in
the tool chain.
Muen separation kernel
======================
Besides updating the Muen port to the latest kernel version as of end of June,
Muen has been added to Genode's automated testing infrastructure. This
includes the revived support for VirtualBox 4 on top of this kernel.
NOVA microhypervisor
====================
The current release extends NOVA to become a Multiboot2 compliant kernel. Used
together with GRUB2, NOVA can now be started on x86 UEFI machines missing
legacy BIOS support (called CSM).
GRUB2 provides the initial ACPI RSDP (Root System Description Pointer) to a
Multiboot2 kernel. The RSDP contains vital information required to bootstrap
the kernel and the operating system in general on today's x86 machines. To
make this information available to the user-level ACPI and ACPICA drivers, the
kernel propagates the RSDP to Genode's core, which - in turn - exposes it to
the user land as part of the _platform_info_ ROM module.
In order to ease the setup of an UEFI bootable image, we added a new image
module to our run-tool infrastructure. The run option 'image/uefi' can be used
instead of 'image/iso' in order to create a raw image that contains a EFI
system partition in a GUID partition table (GPT). The image is equipped by the
new 'image/uefi' module with the GRUB2 boot loader, a GRUB2 configuration, and
the corresponding Genode run scenario. The final image can be copied with 'dd'
to a bootable USB stick. Additionally, we added support to boot such an image
on Qemu leveraging [https://www.tianocore.org - TianoCore's] UEFI firmware.
As a side project, minor virtualization support for AMD has been added to
Virtualbox 4 and to the NOVA kernel on Genode. This enables us to run a 32-bit
Windows 7 VM on a 32-bit Genode/NOVA host on an (oldish) AMD Phenom II X4 test
machine.

0
doc/release_notes-17-11.txt → doc/release_notes/17-11.txt
View File

898
doc/release_notes/18-02.txt Normal file
View File

@@ -0,0 +1,898 @@
===============================================
Release notes for the Genode OS Framework 18.02
===============================================
Genode Labs
After being developed for over a decade, Genode remained a mystery for many
people who looked at the project from a distance as it does not seem to fit
any established category of software. In 2018 - declared as the Year of Sculpt
on our [https://genode.org/about/road-map - roadmap] - this will hopefully
change. Genode 18.02 features the first revision of Sculpt, which is a
Genode-based operating system for general-purpose computing. After being used
as day-to-day OS by the entire team of Genode Labs for several months, we feel
that the time is right to share the system with a broader audience (Section
[Sculpt for Early Adopters]).
One fundamental feature of Sculpt is the ability to install and deploy
software from within the running operating system, which is universally
expected from any modern general-purpose OS. Section
[On-target package installation and deployment] presents Genode's unique
take on the topic of software installation and deployment.
Besides Sculpt, the current release has no shortage of other improvements.
Genode's growing arsenal of 3rd-party software received profound updates and
additions, including VirtualBox, Muen, seL4, several GNU packages, and
libraries. Also the user-level networking stack - including the Linux-based
LxIP stack and our custom NIC-router component - received a lot of attention.
Thanks to the added network driver for i.MX-based hardware, this networking
infrastructure becomes usable on embedded platforms based on this SoC.
Furthermore, the current release continues the cultivation of the Nim
programming language for Genode components.
Sculpt for Early Adopters
#########################
The current release features the first revision of Sculpt, which is a
Genode-based operating system for general-purpose computing. This initial
version is called Sculpt for Early Adopters (EA). Its target audience are
enthusiasts who are already familiar with Genode and are eager to use a
Genode-based operating system on their machines. As outlined on the
[https://genode.org/about/road-map - roadmap], later versions will become
increasingly approachable.
[image sculpt_overview]
Please refer to the official
[https://genode.org/documentation/articles/sculpt-ea - Sculpt documentation]
to step right into the adventure.
On-target package installation and deployment
#############################################
In May last year, we introduced the package-management concept for Genode to
pursue two goals. First, to overcome the naturally limited scalability of
composing Genode systems solely from source. This limit became evident in
complex system scenarios that incorporate a huge amount of 3rd party software.
Thanks to the introduced _depot_ concept and its integration in Genode's
workflow - in particular the run tool - the work of system integration became
much more structured (by caring about packages instead of individual build
targets), robust (by avoiding conditions in run scripts), and quick (by the
accelerated test cycle when using pre-built packages).
The second goal is the ability to update and extend a running Genode system on
the fly. We are happy to have reached this goal with the current release. As
exemplified by the Sculpt scenario, packages cannot only be used as building
blocks for system images but also as subsystems dynamically installed and
deployed on target. Even though installation and deployment are closely
related topics, both involve distinct challenges, which allow Genode to shine.
Installation / update
=====================
In traditional operating systems, the installation and update of system
software is the job of privileged programs. For example, a package manager in
a GNU/Linux system is typically executed with root privileges. This is
troublesome because the functionality of such a program is extremely complex.
In particular it is exposed to the network and has to parse content
originating from potentially untrusted parties. Therefore, potential software
vulnerabilities should be expected. However, in modern OSes, these programs
are just assumed to behave correctly. If this overly optimistic assumption
doesn't hold, the entire system is at risk.
Genode helps us to mitigate this problem by modelling each installation step
as a distinct component composition where each component has a well-defined
and extremely narrow role. The installation is an iterative sequence that
is orchestrated by the so-called download-manager component
(Figure [depot_download]).
[image depot_download]
Initially, the download manager receives a list of content to be installed
into the local depot, which is stored on the file system. The depot may
already be populated with (portions of) this content. In the first step, the
download manager must determine the parts that are missing. To do that, it
does not access the file system directly but instead hands over this task to a
disposable helper component called _depot-query_ that is spawned within a
dynamic init instance. This indirection has two benefits. First, the download
manager is not bothered with the complexity of accessing the file system. It
does not even have any notion of files. Second, the download manager is
effectively shielded from the file system. Should the file system misbehave,
the liveliness of the download manager remains unaffected.
[image depot_download_query_deps]
The depot-query component reports its findings to a report session. The report
eventually reaches the download manager as an updated ROM module. Given the
list of missing content, the download manager has to determine the information
of where to obtain the content from and the public key of the content creator.
This information is contained within the depot. So the download manager issues
another request to the depot-query component in order to obtain it.
[image depot_download_query_url]
Once the depot-query component has responded, the download manager knows what
content to get, where to get it, and how to verify it. To download the
content, it changes the dynamic init instance as follows.
[image depot_download_fetch]
The depot-query component is now gone. Actually, the entire depot has moved
out of sight. Instead, a fresh _fetchurl_ component is spawned. This component
is connected to the network as well as the writeable download directory
_public/_. Internally, fetchurl employs a complex software stack, which
includes the C runtime, curl, libssl, and libssh. Hence, we expect this
component to be vulnerable. Since it is facing the network, we assume that
vulnerabilities are exploitable. In the worst case where the component is
completely in the hands of an attacker, it may write wrong content into the
_public/_ location. But compared to executing curl or wget as root on a
traditional Unix system, the reach of an attack is quite limited. For example,
the mere existence of the download manager remains completely out of view of
fetchurl. However, the content of _public/_ must not be trusted. To reinforce
trust in the downloaded content, the content is accompanied with cryptographic
signatures created by the content creator. Before we touch the content, we
first check its authenticity. To perform this verification step, the download
manager reshapes the dynamic init instance as follows.
[image depot_download_verify]
Note that fetchurl exists no more and network connectivity is cut, effectively
disposing any form of malware that might have infected fetchurl. Next a new
_verify_ component enters the picture. It is configured with a list of content
to check, the signatures of the content, and the public key of the content's
presumed creator. Since it accesses the _public/_ location exclusively, it is
not prone to any potential time-of-check to time-of-use problems during the
verification. Under the hood, the _verify_ component employs a hugely complex
implementation based on GnuPG. It would be naive to fully trust this code.
However, when embedded in our scenario, the reach of a bug is limited because
the verify component has no access to any mutable system state. It could
merely give the wrong answer (which is of course bad but there is no way we
can magically solve this).
Knowing that the downloaded content is indeed the same content as intended
by the creator, it is time for extraction. For this step, the download
manager - again - reshapes the dynamic init instance:
[image depot_download_extract]
This time, both the _public/_ location as well as the trusted _depot/_ are
visible and a new _extract_ component is spawned. As the depot may host
content from multiple sources, which potentially distrust each other, the
content of each content provider resides in a dedicated subdirectory within
the depot. Instead of handing over access to the entire depot to the extract
tool, we mediate the file-system access via a _chroot_ component that limits
the view to the depot-provider's respective subdirectory. In the worst case
where a misbehaving content provider delivers a forged (but correctly signed)
archive to exploit a vulnerability of the extract component, the reach of the
attack remains limited to the content provider's space within the depot.
After the extraction step has completed, the depot is populated with the new
content, which may - in turn - include new dependency information. At this
point, the download manager starts a new iteration. This iterative process
terminates as soon as the depot-query component signals that no content of
the software installation is missing.
The bottom line here is that we are able to use complex and useful software
like curl, libarchive, liblzma, and GnuPG while largely distrusting it. In
contrast to this software that sums up to hundreds of thousand lines of code,
the download manager comprises less than 1000 lines of code. The software
installation procedure described above is implemented by the 'depot_download'
subsystem hosted in the gems repository and illustrated by an equally named
run script. It also forms the basis of the install/update mechanism of the
Sculpt scenario.
Deployment
==========
Once software has entered the system in the form of depot content, the
remaining question is how to turn this content into running subsystems. The
answer is given by the following illustration.
[image sculpt_deploy_runtime]
Like for the installation process described above, the scenario employs a
dynamic init instance that is accompanied by an orchestrating component. The
latter is called _depot-deploy_. The depot-deploy component queries
information from the depot using the same depot-query component that was used
during the installation. Based on the returned _blueprint_ information for the
to-be-deployed subsystems, it generates the configuration for the dynamic init
instance. The subsystems hosted within this init instance access the depot
content via mere ROM sessions as provided by the FS-ROM component. This makes
the use of the depot transparent to the hosted subsystems.
The depot-deploy component is located in the gems repository and accompanied
by a same-named run script. More importantly, it is featured in the deploy
runtime of the Sculpt system.
Base framework and OS-level infrastructure
##########################################
Increased default warning level
===============================
For building Genode components written in C++, the compiler flags -Wextra,
-Weffc++, and -Werror are now enabled in addition to -Wall by default.
If this strict warning level is inapplicable for a given component or
library, it is possible to explicitly disable the strictness in the
respective build-description file by adding the following line:
! CC_CXX_WARN_STRICT =
We adjusted almost all the code of the base, base-<kernel>, os, and demo
repositories to comply with this new warning level. For most components
hosted in the higher-level repositories (libports, ports, dde_*, gems),
the strictness is disabled as of now and will be enabled component-wise
wherever feasible.
While adjusting our code base, we identified the following patterns worth
mentioning:
* A class with virtual functions can no longer publicly inherit base
classes without a vtable. The inherited object may either be moved
to a member variable, or inherited privately. The latter would be
used for classes that inherit 'List::Element' or 'Avl_node'. In order
to enable the 'List' and 'Avl_tree' to access the meta data, the
'List' must become a friend.
* Instead of adding a virtual destructor to abstract base classes,
we inherit the new 'Interface' class, which contains a virtual
destructor. This way, single-line abstract base classes can stay
as compact as they are. The 'Interface' utility resides in
_base/include/util/interface.h_.
* With the new warning level, all member variables must be explicitly
initialized. Basic types may be initialized with '='. All other types
are initialized with braces '{ ... }' or as class initializers. If
basic types and non-basic types appear in a row, it is nice to only
use the brace syntax (also for basic types) and align the braces.
* If a class contains pointers as members, it must now also provide a
copy constructor and assignment operator. In most cases, one
would make them private, effectively disallowing the objects to be
copied. Unfortunately, this warning cannot be fixed by inheriting
our existing 'Noncopyable' class (the compiler fails to detect that
the inheriting class cannot be copied and still gives the error).
For now, we have to manually add declarations for both the copy
constructor and the assignment operator as private class members.
Those declarations should be prepended with a comment like this:
! /*
! * Noncopyable
! */
! Thread(Thread const &);
! Thread &operator = (Thread const &);
In the future, we plan to revisit these occurrences and try to replace
the pointers with references. In the presence of at least one
reference member, the compiler would no longer implicitly generate
a copy constructor. So we could remove the manual declaration.
The following caveats are expected, even if you disable the strictness
in your component:
* If your component has a class called 'Interface', it may collide with
the new 'Genode::Interface' class. You may have to disambiguate the
names.
* The 'Genode::Rpc_client' is no longer a 'Genode::Capability'. Hence,
classes inherited from 'Genode::Rpc_client' cannot refer to a
'Capability' but must refer to 'Genode::Capability'.
* The 'Surface' class is no longer copyable, which led to API
changes of users of this class. E.g., the 'Nitpicker_buffer'
utility does no longer offer accessors for the contained surfaces
but a new 'apply_to_surface' method that takes a lambda function as
argument.
Init
====
Init selects session routes based on the requested service and the client's
label. The latter can be matched as 'label' (exact match), 'label_prefix', or
'label_suffix' (either end of the label matches). With the new version, these
options are complemented with an additional 'label_last' attribute that covers
the prominent case where the last part of the label identifies a requested
resource at the server. A typical example is the routing of a ROM session
based on the name of the requested ROM module.
Reflecting the core log to the application level
================================================
Core records now log messages in a ring buffer and exports this
memory as ROM named 'core_log'. User applications may monitor this ring buffer
and present or transfer the content as appropriate. The example component in
_repos/os/src/app/log_core_ transforms the content into normal log
messages, which may be routed to graphical terminals or stored on
file systems, e.g. by using the fs_log server.
NIC-router improvements
=======================
During the past three months, the NIC router has received several improvements
that were mainly inspired by our daily experience with the component as part
of our Sculpt based working environments.
The most notable new feature is the support for multiple NIC sessions at one
domain. If multiple NIC-session clients connect to one domain, the NIC router
acts as a simple hub between them. I.e., for every packet that is routed to
the domain, each connected session receives a copy of the packet. The same
applies for domain-local packets, meaning packets that target an IP address
inside the IP subnet of the domain they came from. This domain-local
forwarding applies before considering any other routing rules. So, in other
words, it is not possible to route such traffic to another domain.
Furthermore, the logging features of the NIC router were improved. First, the
router is now capable of periodically sending a report via Genode's report
session. This can be activated by adding the new '<report>' node to the router
configuration:
! <config>
! <report interval_sec="5" bytes="yes" config="yes">
! ...
! </config>
So far, the report provides per-domain information about the amount of sent
and received data ('bytes' attribute) and the current IPv4 configuration like
IP address, subnet mask, and gateway address ('config' attribute).
Second, there is a new verbosity option in the '<config>' node:
! <config verbose_domain_state="yes">
When this option is set, the NIC router will output a short message to the log
for each general state change of a domain. Currently, this includes the
IP-configuration state (IP address, subnet mask, gateway address) and the
number of connected NIC sessions. This is a useful addition because the
purpose of the regular verbose option is to give a very deep insight into
almost every activity of the router, which is vital for debugging
sophisticated problems but normally floods the log. Therefore, the regular
verbose option is not viable for complex setups like a Sculpt desktop
environment. In such a context, the new domain-state verbosity is pretty
discreet but already gives a good hint on why, for instance, packets get
dropped despite the routing rules being correct.
Last but not least, the timeout configuration of the NIC router has been
reworked and now allows for a much more precise adaption to the network
environment. The former 'rtt_sec' attribute of the '<config>' node has been
replaced by the following new attributes (default values shown):
! <config dhcp_discover_timeout_sec="10"
! dhcp_request_timeout_sec="10"
! dhcp_offer_timeout_sec="10"
! udp_idle_timeout_sec="30"
! tcp_idle_timeout_sec="600"
! tcp_max_segm_lifetime_sec="30">
Details about the new attributes can be found in the
_os/src/server/nic_router/README_ file. The default values should be
appropriate for the common use case so that specifying them is normally not
necessary.
New watch mechanism for file-system session
===========================================
The file-system session already provided a way for watching files or
directories for changes. However, the original mechanism was arguably hard to
use. In addition to opening the to-be-watched file-system node, the client had
to submit a so-called content-changed request into the session's request
queue. In turn, the server delivered the change notification by acknowledging
this request.
The new mechanism is much less bureaucratic. A file or directory can be
watched by opening a watch handle rather than submitting a 'CONTENT_CHANGED'
packet to the server. Whenever a change happens at a node with an open watch
handle, a CONTENT_CHANGED packet will be sent from the server to the client.
This serializes the registration with other handle operations and separates
I/O handle state from notification handle state.
C runtime
=========
We changed libc's handling of 'clock_gettime' to be explicitly configurable
rather than relying on built-in heuristics. With the new version, the libc
opens a timer session as a time source only if the 'rtc' attribute of the
'<libc>' configuration node is defined. If not configured, 'clock_gettime'
returns 0.
This change may require the adjustment of components that implicitly rely on
the libc as time source. To enable such a component to use relative time
(based on a timer session) but no wall-clock time, one can manually provide a
pseudo real-time clock value as follows:
! <vfs>
! <dir name="dev">
! <log/> <null/> <inline name="rtc">2000-01-01 00:00</inline>
! </dir>
! </vfs>
! <libc stdout="/dev/log" stderr="/dev/log" rtc="/dev/rtc"/>
GUI stack and terminal improvements
===================================
Nit-FB improvements
-------------------
The nit_fb component provides a framebuffer and input service while using the
nitpicker GUI server as back end. The new version adds the 'initial_width' and
'initial_height' attributes, which accommodate the use case where nit_fb is
used in a dynamic fashion like as a client of a window system. Here, the
initial dimensions define the initial window size but - in contrast to the
existing 'width' and 'height' attributes - the actual size can change
afterwards.
Terminal resizing
-----------------
The terminal-session interface gained the ability to propagate resize events
from the server to the client. The new version of the graphical terminal uses
this mechanism to support window resizing as well as dynamically changing the
font size. At the client side, noux has become able to reflect terminal-size
changes to noux applications. Applications based on ncurses (e.g., vim) are
able to gracefully respond to such changes now.
Using chroot to enforce read-only file-system access
====================================================
By placing a chroot component in-between a file-system client and server, the
client's view on the file system can be limited to a specific directory. With
the current release, chroot can additionally be used to restrict a writeable
file-system session to become read-only. This is accomplished by the new
'writeable' attribute of chroot's policy nodes. By default, it is set to "no".
API changes
===========
Noncopyable AVL node/tree
-------------------------
Copying an AVL node generally violates the integrity of the corresponding
tree. To rule out subtle bugs where AVL nodes are accidentally copied, AVL
nodes are no longer copyable.
New 'Buffered_xml' utility
--------------------------
The 'Buffered_xml' utility located at _os/buffered_xml.h_ simplifies the
implementation of dynamically reconfigurable components that need to keep a
verbatim copy of certain parts of their configuration during configuration
updates.
New 'List_model' utility
------------------------
More and more components respond to dynamic configuration updates. For most
components, such updates are quite simple: replace an old internal state by a
new one. But in cases like init, menu_view, or window decorator, a
differential update is in order. Until now, each of these components employed
custom code for this task. As this code is not trivial, a common solution is
preferable. This solution comes in the form of the new 'List_model' utility
located at _base/include/util/list_model.h_. It introduces a light-weight
formalism to feed a component-internal data model from an externally-provided
XML structure.
Dynamically expandable reporter utility
---------------------------------------
In many cases, components that generate reports don't explicitly handle the
situation where the default buffer size of 4096 bytes is exceeded by the
report. This problem is easy to miss because reports are often small at
testing time but become larger when deployed in complex scenarios. In most
cases, the best way to handle an 'Xml_generator::Buffer_exceeded' exception is
upgrading the report session. The new 'Expanding_reporter' that accompanies
the original 'Reporter' in _os/reporter.h_ eases the handling of this common
case.
Languages and runtime environments
##################################
Nim programming language
========================
A new Nim library for constructing Genode servers is now available in the
World repository. This module provides utilities for the asynchronous
session-creation procedure introduced in the
[https://genode.org/documentation/release-notes/16.11#New_session-creation_procedure - 16.11]
release. Some introductory code snippets are provided here for the
adventurous.
An example of server creation using the 'genodeservers' module:
! import romclient, genodeservers
!
! var
! sessionsRom = newRomClient "session_requests"
! # synchronously open a ROM client to the parent
! romContent = sessionsRom.stream.readAll()
! # copy the ROM content to a heap string
! requestsParser = initSessionRequestsParser(romContent)
! # a state machine for parsing 'session_requests' XML
!
! for id, service, label in requestsParser.create:
! # the `create` iterator provider for the parser
! # hides the details of parsing the XML data
! discard txBufSize = requestsParser.argInt "tx_buf_size"
! # extract typed session arguments from the current parser state
! discard label.lastLabelElement()
! # label handling utilities are provided
! if service == "MyService":
! myCreateSessionProc(id, label)
!
This module streamlines the handling of session metadata, but the developer
must still provide hand-crafted wrappers over the C++ methods for managing
RPC objects and passing session capabilities to the parent. Most notoriously
a global pointer symbol, `genodeEnv`, is used to expose the component
environment object. In the future, this will be replaced by a typed object
passed from runtime to an application entry procedure.
! type MySessionCapability {.
! importcpp: "My_session::Session_capability",
! header: "my_session/capability.h".}
! # import a capability type
!
! type MyNativeSessionBase {.
! importcpp: "My_session::Session_rpc_object",
! header: "my_session/rpc_object.h".}
! # import C++ session RPC object
!
! type MyNativeSession = Constructible[MyNativeSessionBase]
! # apply the C++ Constructible template to defer calling
! # the object constructor
!
! proc construct(cppObj: MyNativeSession) {.
! importcpp: "#.construct(*genodeEnv)".}
! # call the C++ constructor, passing the global Genode::Env
!
! proc manage(cppObj: MyNativeSession): MySessionCapability {.
! importcpp: "genodeEnv->ep().manage(*#)".}
! # call a method from the gobal Env, dereferencing
! # thru the Constructible template
!
! type MyNimSessionObj = ref object
! cppImpl: MyNativeSession
! cap: MySessionCapability
! id: SessionId
! # C++ RPC objects are best kept in native
! # reference-counted Nim objects
!
! proc manage(obj: MyNimSessionObj) =
! obj.cppImpl.construct() # call our wrapped constructor
! GC_ref(obj)
! # manually increase the reference count on our session
! # object to prevent the component entrypoint from
! # referencing an RPC object that has been lost and
! # freed from the heap
! obj.cap = obj.cppImpl.manage() # store our capability
!
! proc myCreateSessionProc(id: SessionId): MyNimSessionObj =
! result = new MyNimSessionObj
! # create our object on the heap
! result.manage()
! # construct and manage our RPC object
! result.id = id
! # store the session id from our parent
Procedures for calling Nim code from an RPC object, dissolving
and destructing RPC objects, and managing the session lifetime
are exercises left to the reader.
Updated VirtualBox
==================
Our VirtualBox port got updated from version 5.1.22 to version 5.1.32 in order
to leverage the security updates and improved audio support. Additionally the
boot time of Linux guests got improved by adjusting our custom virtualization
back end.
Libraries and applications
##########################
New trace-logging component
===========================
The new trace-logger component can be used to easily gather, process, and
export different types of tracing data. Furthermore, it marks the next step
towards a user framework that makes access to Genode's manifold tracing
abilities
([https://genode.org/documentation/release-notes/13.08#Light-weight_event_tracing - 13.08],
[https://genode.org/documentation/release-notes/13.11#Improved_event_tracing - 13.11],
[https://genode.org/documentation/release-notes/15.08#Enhanced_tracing_facilities - 15.08])
intuitive and convenient.
The component can filter the available tracing subjects according to session
label policies and thread names. The processing of the tracing data can then
be configured for each selected subject individually, for groups of subjects,
or for all subjects together. The resulting data is exported as log output.
This is an example configuration of the trace logger, which shows the default
value for each attribute (except policy.thread and policy.label):
! <config verbose="no"
! session_ram="10M"
! session_arg_buffer="4K"
! session_parent_levels="0"
! period_sec="5"
! activity="no"
! affinity="no"
! default_policy="null"
! default_buffer="4K">
!
! <policy label="init -> timer" />
! <policy label_suffix=" -> ram_fs" />
! <policy label_prefix="init -> encryption -> "
! thread="worker"
! policy="null"
! buffer="4K" />
! </config>
The most important features so far when it comes to processing the traced
data are:
* Trace CPU activity and affinity ('activity' and 'affinity' attribute),
* Install individual policies for the creation of further tracing data
('policy' attributes) for instance, 'rpc_name' for a log of issued RPC calls),
* Dimensioning the subject-local trace buffers and the frequency of Trace Logger
data examination ('buffer' and 'period' attributes), and
* Configure the session to the Tracing server ('session' attributes).
A comprehensive documentation of the trace-logger component can be found in
_os/src/app/trace_logger/README_. An example of how to use the component is
given through the run script _os/run/trace_logger.run_.
New component for extracting archives
=====================================
The new 'extract' component located at _libports/src/app/extract_ extracts
the content of an arbitrary number of tar.xz archives according to its
configuration. It is used by the depot-download subsystem described in
Section [On-target package installation and deployment]. The component
is accompanied by the run script _libports/run/extract.run_ that illustrates
its use.
New signature-checking tool based on GnuPG
==========================================
The on-target installation of software packages requires a way to verify
cryptographic signatures of downloaded content within a Genode system.
The new 'verify' component located at _ports/src/app/verify_ facilitates the
code of GnuPG to verify detached OpenPGP signatures against public keys.
Since GnuPG depends on libgcrypt and libgpg-error, ports of those libraries
were added to the libports repository. The component comes with the run
script _ports/run/verify.run_ that demonstrates its usage.
Fetchurl component for downloading files
========================================
Fetchurl is a component for downloading files from the network, based
on the curl library. It used to reside in the genode-world repository.
Since it has become a mandatory part of Genode's on-target software
installation mechanism, we have moved it to the _libports_ repository now.
Besides this relocation, fetchurl received a welcome modernization. In
particular, the new version uses the modern socket-fs infrastructure of
the libc instead of relying on the deprecated libc_lwip plugin as a hard-wired
dependency.
New interactive FLIF image viewer
=================================
A simple image viewing application for the FLIF lossless image format was
written from scratch using the FLIF reference decoder library. The viewer can
be used to interactively view a directory of images and supports animation of
GIF-like FLIF files.
Ported 3rd-party software
=========================
With the current release, the following 3rd-party software becomes available
on Genode:
:[https://www.libarchive.org/ - libarchive]: is a library for uncompressing
and extracting various archive formats. It nicely wraps format-specific
libraries like zlib behind a unified and easy-to-use API. The port can
be found in the _libports_ repository.
:[https://lz4.github.io/lz4/ - lz4] and [https://tukaani.org/xz/ - liblzma]:
implement modern compression algorithms as supported by libarchive.
Thanks to Ben Larson for contributing the port of these libraries.
:[https://www.tcl.tk/ - Tcl]: is used as scripting language for various
Genode tools. With the new 'check_abi' tool described in Section
[Automated ABI consistency checks], the Tcl shell 'tclsh' has become
a dependency of the build system. Therefore, we made 'tclsh' available as
noux package. Note, however, that this port comprises solely the
functionality needed for simple scripting.
:[https://flif.info/ - FLIF]: is a library for the Free Lossless Image
Format. Thanks to Emery Hemingway for making it available in the
genode-world repository.
:[https://github.com/json-c/json-c/wiki - JSON-C]:
is a library for processing JSON-formatted data. Thanks to
Johannes Kliemann for contributing the port to the genode-world
repository.
:[https://www.nlnetlabs.nl/projects/ldns/ - Drill (ldns)]:
provides a utility for DNS testing. Thanks to Emery Hemingway for adding it
to the genode-world repository as a side activity of improving Genode's
network stack.
Updated packages for the Noux runtime environment
=================================================
The current release updates the following noux packages: less (version 487),
grep (version 3.1), coreutils (version 8.29), tar (version 1.30), findutils
(version 4.6), which (version 2.21), sed (version 4.4), and bash (version
4.4.18). Thanks to Hinnerk van Bruinehsen for this welcome contribution.
Device drivers
##############
Ethernet-driver for i.MX-based Wandboard
========================================
The current release contains a port of the Linux kernel driver for the
Ethernet card family originally produced by Freescale. We followed our
established approach to tailor an independent device-driver environment (DDE)
for the specific driver. To profit from synergies with the existing drivers of
the _dde_linux_ repository, we took the Linux kernel 4.4.3 as reference.
For now the current version is limited to support the Wandboard Quad as this
is the i.MX-based board that is nightly tested by our infrastructure. The
support of other boards using the same IP core is planned for future releases.
The driver can be found in _dde_linux/src/drivers/nic/fec_. To test the driver,
no further configuration is needed and you can have a look at one of the
automatic network tests, like _lwip.run_, as a reference.
Platforms
#########
Execution on bare hardware (base-hw)
====================================
Thanks to Johannes Schlatow from the TU Braunschweig, the support of the
Zynq-7000 boards by our base-hw kernel got extended. It is now possible to use
all CPU cores instead of only the primary one.
Updated Muen separation kernel
==============================
The Muen SK port has been updated to the latest development version 0.9. The
most notable features and improvements are the Crash Audit facility and support
for MirageOS/Solo5 subjects which may be executed alongside Genode/base-hw.
Thanks to this feature, the Muen project has reached a milestone by
self-hosting the [https://muen.sk] website on a Muen system. Currently, the
network driver is provided by a Linux subject but with some work it should be
possible to replace it with a Genode/base-hw nic_drv in the future.
Further details regarding Muen v0.9 can be found in the project's release
notes [https://groups.google.com/forum/#!topic/muen-dev/FPL9sc4yaBE].
Updated seL4 kernel
===================
Our remaining patches regarding UEFI framebuffer support got integrated into
the upstream codebase of the seL4 kernel. Hence, we updated our seL4 port to
the upstream version containing our patches.
Build system and tools
######################
Package management
==================
The package-management tools introduced last year have become a vital part
of Genode's workflow.
:Package management documentation:
[https://genode.org/documentation/developer-resources/package_management]
Prompted by the development of the on-target installation and deployment
mechanism featured in the current release, the tools received the following
refinements:
:Use of tar.xz as archive format: This change significantly reduces the size of
published depot content compared to the previously used tar.gz format.
:Subdirectories for archive versions:
In the original version of the depot layout, archives were named as
'<archive-name>-<version>'. Hence, the depot - in particular the download
location - had directories that grew in two dimensions. First, when new
archives were added. Second, when new versions of existing archives were
added (usually corresponding to Genode's release cycle). In the mid-term,
this would have resulted in a huge number of directory entries, e.g., in the
_src/_ subdirectory. To avoid this problem, the new version uses the scheme
'<archive-name>/<version>' instead. This way, at the _src/_ level, each
archive has one subdirectory (the number of subdirectories corresponds to
the number of archives). Inside the subdirectory, there is one entry per
version.
:Controlled rebuild of binary archives:
When calling the depot/create tool for a binary archive with 'FORCE=1', the
underlying source archives are re-extracted and the binary archive is
rebuilt. This is usually done after local changes in the source tree to
apply version updates to depot archives as needed. However, the implicit
rebuild is superfluous whenever the source-version remains the same. This is
particular inconvenient when re-creating pkg archives that refer to a large
number of src archives. Here, all binaries referenced by the pkg archive are
rebuilt each time. The new 'REBUILD' argument allows the user to skip
superfluous rebuilds in such situations. Normally, 'FORCE=1' implies
'REBUILD=1'. However, by explicitly specifying 'REBUILD=', existing binary
archives whose versions remain unchanged are kept instead of being rebuilt.
Offline validation of XML configurations
========================================
The _tool/run_ tool now automatically checks configurations against
target-specific XML schemes. Each component may define a configuration
scheme-file in its _target.mk_ file as follows:
! CONFIG_XSD = my_config.xsd
When the run tool checks the configuration of an instance of Genode's init
component, it additionally iterates through all start nodes of this
configuration. For each start node, it checks whether the according component
provides a configuration-scheme file and, if so, applies it to the
configuration inside the start node. This is done recursively. I.e., also the
child configurations of a sub-init of a sub-init ... of the top-level init
are covered this way.
Whenever the run tool detects an error in one of the checked configurations,
it stops and points out the location of the error. By now, there exist
configuration schemes for the init, the NIC router, and the trace logger
components. Our intention is that every component that interprets its
configuration will eventually be accompanied by such a scheme - not only to
validate actual configuration input but also to serve as documentation for
users of the component.
Automated ABI consistency checks
================================
In [https://genode.org/documentation/release-notes/17.02#Genode_Application_Binary_Interface - version 17.02],
we introduced a kernel-agnostic ABI, which ultimately paved the ground for
Genode's package management. For the time being, the ABI is not set in stone.
It is expected to evolve for some time until it hopefully approaches ABI
stability in the mid term. Whenever Genode's API changes, the ABI may be
affected. For example, symbol sizes may grow. Until now, side effects on the
ABI had to be curated manually. In practice, however, such side effects are
too easy to miss. Therefore, the current release adds a mandatory ABI checking
step to the build process. A new _tool/check_abi_ tool is invoked whenever a
shared object is built. It reports flaws in the ABI definition (such as
duplicated symbols) as well as inconsistencies between a shared object and its
ABI.

766
doc/release_notes/18-05.txt Normal file
View File

@@ -0,0 +1,766 @@
===============================================
Release notes for the Genode OS Framework 18.05
===============================================
Genode Labs
The driver behind the release 18.05 is the rapid evolution of the Sculpt
general-purpose OS. Following the initial version from February, which was
targeted at early adopters, the new Sculpt for The Curious (TC) introduces a
much more welcoming and empowering user experience (Section
[Sculpt for The Curious]).
It goes without saying that the interactive and dynamic nature of the Sculpt
scenario puts a lot more pressure on Genode's components compared to static
workloads. For example, Sculpt calls for the dynamic adjustment of user-level
network routing, the dynamic detection and management of partitions and file
systems, the support of USB storage devices from diverse vendors, and a way to
adapt the visual appearance to a great variety of screen resolutions. Most
improvements described below are our responses to these challenges.
That said, the release is not short of new features either. E.g., it features
the initial port of OpenJDK's HotSpot VM for executing Java programs on Genode
directly (Section [Java language support]), improves the support for the NXP
i.MX family of SoCs (Section [NXP i.MX SoC]), and enhances the VFS with new
plugins for copy-on-write and the auditing of file accesses
(Section [New VFS plugins]).
The release is complemented by the annual update of the Genode Foundations
book (PDF), which covers the fundamentals of the framework in great detail
(Section [New revision of the Genode Foundations book]).
Sculpt for The Curious
######################
With Sculpt for The Curious (TC), Genode 18.05 features the second revision of
the Sculpt general-purpose OS. Compared to the initial version for Early
Adopters (EA), it features a new interactive system-management component that
streamlines common tasks like the management of storage devices, or
configuring the network connectivity. The highlights of the new version of the
base system image are:
* Live-customization of almost all aspects of the system,
* The ability to install and run software in memory only,
* Hotplugging of USB storage devices,
* New support for NVMe storage devices in addition to SATA disks,
* Interactive network configuration including Wifi connectivity,
* Interactive management and inspection of storage devices and partitions,
* The option to host a complete and customized Sculpt installation on a
USB stick,
* Automated on-demand installation of software packages with visual feedback,
* Scalable fonts that are automatically adjusted to the screen resolution, and
* UEFI boot supported by default.
The base image is extensible by downloadable software packages that may
originate from different sources, safeguarded by cryptographic signatures.
It contains several example subsystems as a starting point:
* Basic GUI components like a window manager, a scalable backdrop, a
font server, and a simple software-rendering demo,
* A light-weight noux runtime for executing command-line-based software
such as GNU coreutils, bash, and vim.
* A package for downloading the installer and a suitable virtual-machine
configuration for Debian Linux,
* VirtualBox running Debian Linux,
* An example for running libretro-based games,
* A disposable VM that runs Firefox on TinyCore Linux, executed either in
VirtualBox or the light-weight Seoul virtual-machine monitor,
* A Qt5-based text editor.
Please refer to the updated
[https://genode.org/documentation/articles/sculpt-tc - Sculpt documentation]
to explore Sculpt TC.
The Sculpt version included with the current release requires the user to
build a boot image by hand. Following the steps described in the
documentation, this procedure takes a few minutes. We plan to provide
downloadable boot images a few weeks down the road once Sculpt TC received
intensive day-to-day testing by the early adopters. Your feedback is very
welcome!
New revision of the Genode Foundations book
###########################################
The "Genode Foundations" book received its annual revision, which reflects
the evolution of the framework over the past year. Specifically, the changes
since the last year's edition are:
: <div class="visualClear"><!-- --></div>
: <p>
: <div style="clear: both; float: left; margin-right:20px;">
: <a class="internal-link" href="https://genode.org">
: <img class="image-inline" src="https://genode.org/documentation/genode-foundations-title.png">
: </a>
: </div>
: </p>
* Changed boot-loader infrastructure on PC hardware
* Package management
* Structural changes of Genode's custom base-hw kernel
* API improvements: Unicode handling, support for XML-based data models,
timeout-handling API
: <div class="visualClear"><!-- --></div>
To examine the changes in detail, please refer to the book's
[https://github.com/nfeske/genode-manual/commits/master - revision history].
Storage infrastructure
######################
VFS library and plugin interface
================================
The VFS (Virtual-File-System) library was expanded to meet new requirements
for the Sculpt scenario. The traditional file-system medium for component
state and configuration sculpting is the *ram_fs* server, but with the
limitation that files stored in the server are ephemeral. Any changes to
the initial state are lost when a system is shut down or the *ram_fs* server
is restarted. Now that persistent storage is usually served by a VFS plugin
hosted by the VFS server, it was a natural progression to introduce a means
for indicating VFS changes with 'File_system' session notifications. To this
end the VFS server was amended to send session notifications, and notification
support was added to the Rump and FatFs VFS plugins, allowing Ext2 and FAT
file-systems to host dynamic component state and configuration information.
Using the VFS for serving font data produced from files stored in the VFS made
it practical to allow VFS plugins to introspect the file system. Plugins now
have the means to access arbitrary paths from the file-system root or they may
host and expose their own internal file systems.
While the core of the VFS library is small compared to contemporaries in other
operating systems, the moment came to promote the VFS from a static to a
shared library. Components that use the C runtime have always loaded the VFS
dynamically as a subsystem of _libc.lib.so_, but native components carried the
bulk of its implementation. The VFS library is now provided as a shared
library and is included with the front-end server in the _src/vfs_ depot
archive. This change affects components that have been rebuilt against the
shared library but do not have their ROM policies updated to allow access to
the _vfs.lib.so_ ROM.
New VFS plugins
===============
File-system introspection has made two additional plugins possible, the *audit*
and *cow* plugins.
The *audit* plugin logs VFS paths as they are accessed to a dedicated LOG
session. This is useful for finding the files required by third-party
components without relying on documentation or auditing source code.
The *cow* plugin emulates copy-on-write behavior by copying the contents of
files lying in a read-only path to a read-write path as they are opened. This
plugin is considered a proof-of-concept and under-performing, but opens a way
of experimenting with seeding user-managed file-systems from immutable
file-system archives.
Plugins of this kind are most appropriately instantiated in the VFS server
with policies to restrict the intended components into paths provided by the
plugins. This prevents a component from escaping the effect of the plugin. An
example of "auditing" a libc component follows:
! <start name="audit_fs">
! <binary name="vfs"/>
! <config>
! <vfs>
! <dir name="data"> <!-- source files -->
! <tar "data.tar"/>
! <ram/>
! </dir>
! <dir name="audit"> <!-- virtual path that captures /data -->
! <audit path="/data"/>
! </dir>
! </vfs>
! <!-- route into virtual audit path -->
! <policy label_suffix="audit" root="/audit" writeable="yes"/>
! </config>
! </start>
!
! <start name="app">
! <config>
! <libc stdout="/log" stderr="/log"/>
! <vfs>
! <log/>
! <fs label="audit"/>
! </vfs>
! </config>
! </start>
Improved disk-partition discovery and access
============================================
The 'part_blk' component, which parses the partition table on a block device
and provides access to each partition through a block session, was extended to
make it easier to implement a management component on top of it. It now
features additional attributes in its report. For one the block size of each
partition as well as the type of the file system on the partition are
reported. The file system probing implementation is minimal and only contains
file systems that are commonly used on Genode systems, i.e., FAT32 and Ext2.
Furthermore, on GPT formatted disks, each partition has an 'expandable'
attribute that contains the number of blocks by which the partition can be
grown. The following exemplary report illustrates the adjustments:
!<partitions type="gpt" total_blocks="500118192" gpt_total="500118125" gpt_used="302254080">
! <partition number="1" name="BIOS boot partition"
! type="21686148-6449-6e6f-744e-656564454649" guid="db0701aa-02ae-474d-92d0-82738bfce5d2"
! start="2048" length="2048" block_size="512"/>
! <partition number="2" name="EFI System"
! type="c12a7328-f81f-11d2-ba4b-00a0c93ec93b" guid="74e43226-2afb-4575-bdda-83bf72f5a6e7"
! start="4096" length="262144" block_size="512" file_system="FAT32"/>
! <partition number="3" name="GENODE"
! type="0fc63daf-8483-4772-8e79-3d69d8477de4" guid="a950091d-87ba-4800-85bf-7b6a58abe6d5"
! start="235147264" length="67108864" block_size="512" file_system="Ext2"
! expandable="197862064"/>
!</partitions>
The heuristics of how the component probes the partition table were also
loosened. Instead of explicitly enabling support for GPT, the component will
now always try to parse the MBR as well as the GPT. It will bail out if both
are considered valid since using GPT/MBR hybrid tables is not supported and it
should be up to the user to make an educated decision. In cases where there is
no partition table, a 'partitions' report of 'type="disk"' will be generated
in which the complete disk is presented as partition number '0'. This is
needed as compatibility fallback for Sculpt EA installations.
Creating and modifying GUID partition tables
============================================
Part of the enhancements of Sculpt TC is the ability to manipulate the block
device used by Sculpt. We implemented a component called 'gpt_write', which
can create and modify a GPT and its entries. It considers alignment
constraints to make better use of 512e devices. It will, however, not perform
any boundary checking. It does not handle overlapping partitions and only when
applying a partition, it makes sure that the partition will fit. The following
configuration illustrates its operation:
!<start name="gpt_write">
! <resource name="RAM" quantum="2M"/>
! <config verbose="yes" initialize="yes" align="4K">
! <actions>
! <add entry="1" type="BIOS" label="GRUB BIOS" start="2048" size="1M"/>
! <add entry="2" type="EFI" label="EFI System" start="4096" size="16M"/>
! <add entry="3" type="Linux" label="GENODE" start="36864" size="128M"/>
! <add type="BDP" label="FAT32 Data" size="max"/>
! <delete entry="1"/>
! <delete label="FAT32 Data"/>
! <modify label="GENODE" new_label="GENODE*" new_size="max"/>
! </actions>
! </config>
!</start>
Please read _repos/gems/src/app/gpt_write/README_ for more detailed information
on how to use the component and feel free to check out the run script
_repos/gems/run/gpt_write.run_.
User-level networking
#####################
NIC router
==========
The NIC router has received major improvements that were mainly motivated by
our daily experience with the Sculpt scenario where the router serves as NAPT
component in front of the virtual machines that host our work OS's. In this
role, it is subject to a permanent load driven by real-world tasks.
Furthermore, it has to have a user interface that makes it a pleasant
experience to deploy in a dynamic environment. This led to our primary goal:
We had to overcome the need to restart the NIC router, and thereby all
components that depend on it, whenever its configuration changes and while
doing so, not to interrupt the communication of its client unnecessarily.
We managed to make the NIC router fully re-configurable at runtime in a way
that it always tries to keep as much state information as possible throughout
the process. This means that network communication going through the NIC
router is not affected by a configuration update unless the configuration
change affects parts that were involved in an existing communication channel.
One prerequisite for this feature was that NIC session clients can connect at
any time to the NIC router regardless of whether there is a matching domain
for the session or not. As long as a session has no domain, the NIC router
does not send any packet to it and drops all packets coming from it. But, at
least, the session and the corresponding client component stay alive, even if
their already assigned domain disappears with a new configuration.
At the uplink, in contrast, the lifetime of the session remains bound to the
lifetime of the domain. The uplink domain-tag received a new attribute
named 'label' (only considered at the domain-tag of the uplink). It denotes
the label of the uplink session. With these two particularities of the uplink
domain, one can now easily switch between different NIC session servers. The
NIC router will close and request the corresponding NIC session with the
current 'label' value if the 'domain' node is removed/added or the label
changes. Thereby, the NIC router can now be used to dynamically switch between
network interfaces like wireless and wired adapters.
Furthermore, we improved the NIC router's ability to handle DNS server
information. Domains can wait for the DNS server info of the DHCP client of
another domain. This is done with the new attribute 'dns_server_from' in the
'<dhcp_server>' tag. Each time the DNS server info of the remote domain
changes, the DHCP server with the 'dns_server_from' attribute will toggle the
link state of each session at its domain. This can be used by clients as a
hint to request their DHCP info anew from the NIC router and thereby receive
the updated DNS server information.
When it comes to protocols, the most notable change is that the NIC router now
also supports routing and NAPT for ICMP. With the new '<icmp>' sub node of the
'<domain>' tag, ICMP routes to other domains can be created. Instead of ports,
the ICMP IDs are used for NAPT. Similar to the 'udp-ports' and 'tcp-ports'
attributes, the size of the ID space for each NAPT client is configured via
the new 'icmp-ids' attribute in the '<nat>' tag.
Last but not least, the following small features were also added to the NIC
router:
:Attribute 'verbose_packets' for the '<config>' and the '<domain>' node:
Toggles the logging of most important protocol header fields globally or
domain-locally. The 'verbose' attribute does not affect this kind of debug
output anymore.
:Report DNS server info:
If the 'config' attribute in the '<report>' node is enabled, the NIC router
will now also report the DNS server info for each domain.
:Attribute 'config_triggers' in the '<report>' node:
Toggles whether the NIC router immediately sends a report whenever the IPv4
configuration of a domain changes, regardless of any timeouts.
:IPv4 point-to-point support:
If a domain receives an IP configuration with a subnet mask of
255.255.255.255 it will switch to point-to-point IPv4 (requires a valid
gateway address at the domain).
:ICMP destination unreachable on non-routable packets:
The NIC router now responds with an ICMP "destination unreachable" packet to
packets that are not routable at an interface with a domain.
For more information, have a look at the _os/src/server/nic_router/README_
file. Examples can be found in the run scripts
_dde_linux/run/nic_router_uplinks.run_,
_libports/run/nic_router_dyn_config.run_, and _os/run/ping_nic_router.run_.
NIC dump
========
The output level of the NIC dump component can now be configured per protocol
by using the protocol names as attributes: 'eth', 'arp', 'ipv4', 'dhcp',
'udp', 'icmp', and 'tcp'.
The available debug levels are:
:no: Do not print out this protocol.
:name: Print only the protocol name.
:default: Print a short summary of the most important header values.
:all: Print all available header values.
Additionally, you can set a default debug level for protocols that are not
configured using the 'default' attribute.
For more information, please refer to _os/src/server/nic_dump/README_.
GUI stack
#########
With Sculpt becoming more and more end-user oriented, Genode's GUI stack came
into focus. It was time to reconsider several interim solutions that worked
well in the past but would not scale up to a modern general-purpose OS. Two
concrete examples are the support of scalable fonts and Unicode characters. In
the past, Genode used to restrict textual output to the Latin-1 character set
and employed pixel-based fonts only. The current release overcomes these
limitations by featuring completely new text-output facilities.
UTF-8 support and improved text rendering
=========================================
The UTF-8 text encoding overcomes the severely limited code-point range of the
ASCII and Latin-1 character sets by representing characters by a varying
number of bytes. Today, UTF-8 is generally considered as the standard encoding
for text. The new UTF-8 decoder at _os/util/utf8.h_ clears the path for
Genode's native GUI components to follow suit. The first beneficiary is
Genode's graphical terminal, which has become able to display Unicode
characters and pass user input as UTF-8-encoded data to its terminal-session
client.
Terminal enhancements
=====================
Speaking of the graphical terminal, the current incarnation got a welcome
overhaul. First, we reduced its complexity by removing obsolete features like
built-in keyboard-layout handling, which are no longer needed when combining
the terminal with our modern input-filter component. Furthermore, the terminal
has become dynamically resizeable, forwarding screen-size changes to the
terminal client. Should the client be a Noux runtime, such a change is
reflected to the running application as a SIG_WINCH signal. The application -
e.g., Vim - responds to the signal by requesting the new terminal size.
Finally, the terminal protocol was changed from 'linux' escape sequences to
'screen' escape sequences in the anticipation of making the terminal more
flexible in the future.
Text rendering
==============
Throughout Genode, many GUI components reused the text-output utilities
of the nitpicker GUI server. These utilities, however, relied on a simple
pixel font format. To make the text output more flexible, nitpicker's text
painter located at _nitpicker_gfx/text_painter.h_ has been replaced by a
completely new implementation that decouples the font format from the
glyph rendering and takes UTF-8 strings as input. In the process, the glyph
rendering got a lot more sophisticated, supporting horizontal sub-pixel
positioning and filtering.
Font-format support
===================
To remove the omnipresent use of fixed-size pixel fonts throughout Genode,
the following new components entered the picture:
First, the new 'ttf_font' library implements nitpicker's font interface by
using the TrueType renderer of the STB single-header library.
Second, the new 'vfs_ttf' VFS plugin uses the 'ttf_font' library to export a
rendered TrueType font as a virtual file system. The various font properties
as well as the actual glyph images become accessible as regular files. This
way, an application that needs to draw text can read the glyph data directly
from its VFS instead of depending on a font-rendering library.
Third, the new 'Vfs_font' utility located at _gems/include/gems/vfs_font.h_
implements nitpicker's font interface by obtaining the glyphs from the
component-local VFS. It is complemented by the 'Cached_font' utility, which
implements an LRU glyph cache.
With this infrastructure in place, several existing GUI components could
be updated, most prominently the graphical terminal and the menu-view
widget-rendering engine. By facilitating the VFS as interface for propagating
glyph data, components no longer need to manage fonts and their configuration
individually. They just access their VFS. When integrating the component into
a scenario, one can decide whether to mount a font-rendering library directly
at the component, or - alternatively - route a file-system session to a
central font server. The latter is just a regular VFS server with the fonts
mounted as pseudo file systems. Since the glyph renderer is a VFS plugin, it
could be replaced by another implementation in the future without touching any
component.
Modernized API for input-event processing
=========================================
Genode's input-session interface changed very little over the years. Even
though it received evolutionary enhancements from time to time, its design
resembled a traditional C-style interface from the medieval era. We found that
the interface left too much room for interpretation. In particular, the meta
data per event type was defined in a rather ad-hoc way, which raised
uncertainties. For example, is a button-press event accompanied with a
positional value or not? To remove these uncertainties, the current release
replaces the 'Input::Event', with a new implementation that facilitates a safe
way of accessing event meta data. Besides this design change, there is one
noteworthy semantic change as well. With the new interface, symbolic character
information are provided along with their corresponding press events rather
than as distinct events, which - according to our practical findings - greatly
simplifies the consumer side of the 'Input::Event' interface.
Improved keyboard-focus handling
================================
The nitpicker GUI server multiplexes one screen among multiple GUI clients in
a secure way. One aspect remained underdeveloped so far, which is the keyboard
focus handling. Nitpicker's 'Session:focus' call previously triggered a one-off
focus change at call time. This focus change did not pass the same code paths
as a focus change triggered by a "focus" ROM update, which led to
inconsistencies.
The new version changes the implementation of 'Session::focus' such that the
relationship of the caller and the focused session is preserved beyond the
call time. Whenever the calling session is focused in the future, the
specified session will receive the focus instead. So 'Session::focus' no
longer represents a single operation but propagates the information about the
inter-session relationship. This information is taken into account whenever
the focus is evaluated regardless of how the change is triggered. This makes
the focus handling in scenarios like the window manager more robust.
Device drivers
##############
NVMe storage devices
====================
Since NVMe devices have become common in contemporary systems, it is time to
provide a driver for such devices on Genode. With this release, we introduce a
component that is able to drive consumer-grade NVMe storage devices, i.e.,
there is no support for namespace management or other enterprise-grade
features. For now, to keep things simple, the driver uses the device in an
old-fashioned way and uses only one I/O queue with at most 128 entries. That
is to say it does not exploit the parallelism necessary to unlock the full
potential of NVMe storage. Nonetheless, it performs well. The following
snippet illustrates its configuration:
!<start name="nvme_drv">
! <resource name="ram" quantum="8M"/>
! <provides><service name="Block"/></provides>
! <config>
! <report namespace="yes"/>
! <policy label_prefix="client1" writeable="yes"/>
! </config>
!</start>
The component will generate a report, which contains all active namespaces, if
reporting is enabled by setting the 'namespace' attribute of the '<report>'
node to 'yes'. A report may look like the following example:
!<controller model="QEMU NVMe Ctrl" serial="FNRD">
! <namespace id="1" block_count="32768" block_size="512"/>
!</controller>
For an example on how to integrate this component, please have a look at the
_repos/os/run/nvme.run_ script.
While implementing the NVMe driver, a new component for testing block-sessions
was used. In contrast to the already existing 'blk_bench' and 'blk_cli'
components, it features a variety of different test patterns, which can be
selected in its configuration and can be used to test a block component more
thoroughly. For more information please refer to
_repos/os/src/app/block_tester/README_
NXP i.MX SoC
============
We extended the Linux kernel driver port for Ethernet cards found in NXP i.MX
SoC, which was introduced in the previous release. Now does it not only
support i.MX6Q SoC based boards like the Wandboard, but the i.MX53 and i.MX6SX
SoC as well. The new driver was successfully tested with the i.MX53 Quick
Start Board and the Nitrogen6 SOLOX. The latter board even contains two
Ethernet cards. But due to technical limitations of the board design, the same
driver instance has to be used for both cards. Currently, the driver is
tweaked to run on different boards via its configuration ROM. When no
configuration is provided, it appropriates the values for successfully
executing on the Wandboard. The following is an example configuration for the
i.MX53:
! <config>
! <card name="fec0" type="fsl,imx25-fec" mii="rmii" irq="87" mmio="0x63fec000"/>
! </config>
As a side effect of enabling networking on the Nitrogen6 SOLOX, support for
GPIO based signals has been added to the framework too. The existing GPIO
driver for i.MX53 SoC got extended to additionally support the i.MX6 family.
There are some known limitations when using different drivers like Ethernet
and SD-card drivers on the Wandboard right now. At the moment, those drivers
adjust clock parameters and I/O pin configurations independently from each
other, which can lead to inconsistencies. We plan to address those issues with
the implementation of a platform driver for the i.MX6 SoC family.
Improved USB-storage driver
===========================
We improved the stability of the USB-storage driver (usb_block_drv) and
made it compatible with a lot more devices as the driver has become a pivotal
ingredient of the Sculpt scenario. Due to the changes, the way the driver
operates has changed. On the one hand, now it first tries to use 10-byte
Command Descriptor Blocks (CDB) in its SCSI layer and will only switch to
16-byte CDBs when it encounters a device whose blocks cannot be completely
accessed via the former descriptor size. On the other hand, because some
tested devices stopped working after issuing a USB device reset, the reset was
made optional. By setting the 'reset_device' attribute in the '<config>' node
to 'yes', the driver is instructed to perform the USB device reset.
Libraries and applications
##########################
Packaged Qt5 framework
======================
We created package recipes for all previously ported Qt5 libraries and their
dependencies and adapted the run scripts accordingly. Please note that the
host tools needed for building Qt applications (moc, rcc, uic) are not built
automatically anymore, but need to be built and installed manually with the
new 'tool/tool_chain_qt5' script.
Java language support
=====================
Over the course of the past year, we started to look into Java support for
Genode with the ultimate goal of porting an existing Java Virtual Machine
(JVM), which translates and executes Java byte code, to Genode. After
investigating possible JVM candidates, it became obvious that
[https://openjdk.java.net - OpenJDK] is the only viable option when looking for
a functional, maintained, feature complete, and open-source Java SDK.
Therefore, we decided upon OpenJDK version 9 and started to port OpenJDK's
HotSpot virtual machine.
In the first step, we followed the approach to enable HotSpot's internal
Just-in Time (JIT) compiler, which translates byte code into machine code and
is the option with the most to offer performance wise. But we also wanted
support for ARM platforms and soon realized, there was almost no JIT compiler
support for ARM other than for Linux. The Linux version is deeply integrated
into the Linux system libraries (e.g., glibc), which makes it very hard to
bring the compiler onto Genode. For example, Genode uses FreeBSD's libc and
that would now have to offer glibc semantics.
After additional research, we found the so-called interpreter version of the
HotSpot VM. This version does not compile byte code, but interprets and
emulates the code at runtime. It is of course slower than the JIT compiler
version, but also machine-architecture independent, so the same HotSpot VM can
be compiled for x86 and ARM platforms. With the JVM running on Genode, we
added networking and file-system access support via Genode's VFS layer. Note,
there is no graphical toolkit support as of now, but most standard library
classes should work. Also, the byte code has to be compiled on a different
host system (e.g., Linux, *BSD) as of now, since we did not bring the Java
compiler to Genode.
To give Java a spin, a run script can be found under _ports/run/java.run_.
Ada language support
====================
Support for components and libraries written in the Ada/SPARK programming
language experienced a rework with the final goal of seamless integration with
the base framework. We added a new _ada_ library, which contains a (currently
minimal) runtime taken from the sources of our GCC port and thus is always
consistent with the tool chain in use. It is built as a shared library
_ada.lib.so_ that needs to be added to the list of boot modules.
The example in _libports/src/test/ada_ showcases the implementation of an Ada
component using a custom library _test-ada_, which is also implemented in Ada.
Seoul VMM on NOVA
=================
The Seoul/Vancouver VMM - introduced to Genode with release 11.11 - received
some renovations to be able to run recent Linux VMs. Namely the output of the
guest during early boot is now visible and the network models got revised.
Additionally, the Seoul VMM has been packaged and can be used in Sculpt.
Ported software
===============
The [https://dnsprivacy.org/wiki/display/DP/DNS+Privacy+Daemon+-+Stubby - Stubby]
DNS daemon has been ported to begin experimentations with DNS as a native
service. There is a tendency for DNS configuration frameworks to diverge
between operating systems and releases, an inconvenience that is magnified
when maintaining virtual machines. Name-server configuration via DHCP has been
the only constant, so hosting DNS natively and configuring virtual-machines
with the *nic_router* DHCP server presents itself as a viable solution to the
guest resolver quagmire. Expect DNS services in later Sculpt releases.
Platforms
#########
Accessing PCI via ECAM/MMCONF
=============================
The platform driver on x86 is trusted with guarding access to PCI
devices. Up to now, I/O ports have been used to configure the PCI subsystem.
On modern x86 architectures, PCI devices can be configured by using Memory
Mapped I/O (MMIO). This method was introduced with PCI Express and is called
Enhanced Configuration Access Mechanism (ECAM). For Each PCI device a separate
4 KiB MMIO page exists to serves as the configuration interface between OS and
PCI device.
The exact location of all the 4K MMIO pages of the PCI devices is machine
specific and must be determined during the bootstrap phase. The ACPI driver on
Genode is in charge of this procedure and reports the location of the
ECAM/MMCONF region to the platform driver via the 'acpi' ROM.
Besides using a modern PCI interface, switching to ECAM/MMCON served to ease
the execution of Genode/hw on top of the Muen separation kernel.
Kernel-agnostic platform-information handling
=============================================
Up to now, special kernel-specific information was propagated to components
such as Virtualbox, the Seoul VMM, and the timer by reusing the
kernel-provided data structures. For Genode/NOVA, the hypervisor info page
(HIP) was exported as an ordinary Genode ROM. With the rise of Sculpt and the
packaging of components in a - as far as possible - kernel-independent way,
the propagation of kernel-specific information became a stumbling block.
With this release we abandon the 'hypervisor_info_page' ROM of Genode/NOVA and
replace it with a Genode ROM called 'platform_info'. The 'platform_info' ROM
is planned to contain solely information about the host hardware, which may
not be gathered otherwise by Genode components. In the current state it
contains information required by VMMs, namely whether AMD SVM or Intel VMX is
available and usable. Additionally, the ROM contains information about the
frequency of the time stamp counter.
Updated seL4 kernel to version 9.0.1
====================================
Thanks to Hinnerk van Bruinehsen, the seL4 version used by Genode has been
updated to 9.0.1.
Updated Muen separation kernel
==============================
With the addition of memory-mapped access to the PCI config-space in Genode,
base-hw subjects on Muen now only see the effectively assigned physical
devices. This makes it possible to run Genode in parallel with other subjects
and to pass-through different PCI devices for each instance.
The Muen update also brings a much simplified subject info structure plus some
tweaks to the Muen system policy XML format to facilitate easier integration
of new hardware platform specifications.
Build system and tools
######################
Validating 3rd-party code downloads via SHA256
==============================================
This release removes support for verifying source code of third-party ports
with the SHA1 hash algorithm. Last year, SHA1 was banished as a credible
cryptographic hash function after the demonstration of a full collision
attack. Since the
[https://genode.org/documentation/release-notes/14.05 - 14.05 release],
port files have been verified using SHA1, this release replaces all file
digests with SHA256 digests. Any port definitions maintained in external
repositories are required to make these replacements as well. No collisions
have been discovered against source code archives but nonetheless there is an
obligation to widen our margin of safety.
Creating GPT-based disk images by default
=========================================
Up to now Genode's run tool was able to create x86 bootable images in three
flavours:
* Either as ISO bootable by BIOS legacy - 'image/iso', or as
* GPT partitioned disk image only bootable by UEFI - 'image/uefi', or as
* MBR partitioned disk image only bootable by BIOS legacy - 'image/disk'.
With Sculpt came the demand to have a single image type that is in principle
bootable by both UEFI and BIOS legacy. Additionally with Sculpt, we began to
prefer working with GPT partitioned devices.
In the light of the new demands, we changed the 'image/disk' run tool support
to create a GPT partitioned disk image bootable by a legacy BIOS and by UEFI.

1075
doc/release_notes/18-08.txt Normal file
View File

File diff suppressed because it is too large Load Diff

0
doc/release_notes-18-11.txt → doc/release_notes/18-11.txt
View File

836
doc/release_notes/19-02.txt Normal file
View File

@@ -0,0 +1,836 @@
===============================================
Release notes for the Genode OS Framework 19.02
===============================================
Genode Labs
In our [https://genode.org/about/road-map - road map] for 2019, we stated
our goal to make Genode more relevant and appealing for a broader community.
The current release takes a big leap towards that goal: It opens up Sculpt
OS for 3rd-party software providers, introduces a federated blogging
platform about Genode-related topics, and makes the world's
[https://www.tiobe.com/tiobe-index/ - most popular] programming language
(Java) available to Genode users.
With the 4th stage of the evolution of Sculpt OS - themed as "community
experience" (CE) - Genode's custom general-purpose OS introduces a novel and
simple way for users to discover software, and for software providers to
announce software. There is no middle man like an app store or a distribution!
We hope that this federated model of software provisioning and deployment will
have a vitalizing effect on the community around Genode. On a practical level,
the interactivity of the new version is a playful and fun experience. Section
[Sculpt OS as a community experience (CE)] gives a rough overview about
*Sculpt CE*. A ready-to-use disk image will be released mid of March.
When speaking of "software providers", we don't think of anonymous
repositories. Software - and Free Software in particular - is developed and
provided by individuals after all. So in our federated way of software
distribution, we want to highlight this individuality. For this reason, we
launched a new blogging platform for Genode-related stories called
[https://genodians.org - Genodians.org], which gives everyone who is
enthusiastic about Genode - users and developers alike - a platform to express
ideas, announce software, or share practical tips and tricks. As explained in
the [https://genodians.org/nfeske/2019-01-07-welcome - initial posting],
*Genodians.org* is - in the spirit of Sculpt's software distribution model -
also organized in a federated way. The content is hosted and remains
completely under control by the respective authors. The website merely
aggregates and presents the content. It goes without saying that Genodians.org
is based on Genode. The system image that contains the entire web appliance -
from the kernel over the content management to the web server - is contained
in a 8 MiB disk image.
Section [Genodians.org as a showcase of a Genode-based web appliance] goes
into more detail.
We identified the support of popular programming languages as one key aspect
to attract a broader community. With the current release, our port of
*OpenJDK* (Section [Java]) for both 64-bit x86 and 32-bit ARM has reached a
mature state suitable for the creation of web services. This paves the way
towards quite exciting new system creations, like the *Boot2Java* system
presented in Section [Showcase of a Java-based network appliance].
On the account of programming-language support, we are happy to announce
a vastly improved runtime support for *Ada* and *SPARK* (Section
[Ada and SPARK] as well as the initial support for OCaml (Section [OCaml]).
Besides the work on shiny features, the current release cycle included a
profound *spring cleanup* described in Section
[Base framework and OS-level infrastructure].
It thereby finalizes the huge API modernization initiated almost three years
ago.
Sculpt OS as a community experience (CE)
########################################
When we laid out the road map for Sculpt OS a little more than one year ago,
we envisioned four stages of development. Sculpt EA was geared towards
die-hard early adopters facilitating the live editing of the system using a
text editor. Sculpt TC was targeted at "the curious" and included a graphical
user interface for common administrative tasks. Sculpt VC introduced the
visual composition of the system using an interactive graph. Even though
Sculpt already allowed for the installation of components from different
software providers at this stage, it offered no simple means to discover
software and the installation still required the manual editing of "launcher"
text files.
The final - community experience - stage of the plan ought to foster the
federated provisioning of software. Software providers should have the ability
to announce new packages. Conversely, users should be able to "subscribe" to
such announcements, similar how one would subscribe to an RSS feed. Once
software is discovered in this way, it should take only a few clicks to
install and integrate it into the running Sculpt system. No command-line
interface should stand in the way of discovery.
Sculpt CE is the realization of this idea.
For Sculpt users, the discovery starts at the '+' menu. In contrast to Sculpt
VC where the menu offered a rather long list of components to choose from, the
menu now offers nothing to be excited about.
[image sculpt_ce_menu]
However, at the very bottom, there is a new menu entry "Depot ...", which
leads to a sub menu with a list of enabled software providers. If connected to
the network, it also shows an entry named "Selection ...".
[image sculpt_ce_menu_depot]
The "Selection" sub menu allows the user to enable software providers.
[image sculpt_ce_selection]
When clicking on one of the checkboxes, the package index of the provider is
downloaded and the checkbox is highlighted. In the example, "nfeske" is
already enabled.
By clicking on the top-left triangle, one can always go one menu up. Back in
the depot menu, selecting a software provider will now display the index of
available packages. When selecting a new package, it can be installed via a
single click:
[image sculpt_ce_install]
Once the package is installed, the menu takes the user to a dialog for
integrating the package as a new component into the running Sculpt system. For
each resource required by the package, the user can decide how to connect it.
For example, the file system to be mounted at _/config/_ of a fresh
noux-system instance can be routed to any file system service the user wishes.
The noux-system won't know which file system is selected. It will just work
with it.
[image sculpt_ce_routes]
As soon as all routes are defined, the dialog presents a button for adding the
component to the system.
Using this interactive work flow, the discovery of software and its
integration becomes a rather playful process.
To make a once composed system permanent, one can use the inspect window to
copy the _/config/managed/deploy_ file to the _config/19.02/_ directory of your
Genode partition. This way, the deployment configuration will take effect
immediately at boot time.
Genode 19.02 comes with Sculpt CE included. As usual, we will take a bit
of time following the release for thorough testing and refining before
announcing an updated disk image mid March. We would greatly appreciate
your feedback during this phase! For baking a fresh Sculpt image, please
refer to the
[https://genode.org/documentation/articles/sculpt-vc - documentation]
of Sculpt VC, but using the Git tag '19.02' instead of 'sculpt_vc'.
Announcing software packages
----------------------------
The community experience of Sculpt CE will ultimately depend on the
participation of software providers. To become listed in the selection dialog
mentioned above, you may consider including your public key and download
location at the Genode repository at
[https://github.com/genodelabs/genode/tree/master/depot - /depot/].
For the announcement of packages, a software provider can publish so-called
"index" files for a particular Sculpt version in the software provider's
depot. E.g., the index of packages supported on Sculpt 19.02 would be located
at _index/19.02_ within the depot. Like any other depot content, index files
are digitally signed. An example index for the software provider "nfeske"
would look as follows:
! <index>
! <index name="GUI">
! <pkg path="nfeske/pkg/sticks_blue_backdrop/2019-02-22"
! info="default desktop background"/>
! <pkg path="nfeske/pkg/themed_wm/2019-02-26"
! info="ready-to-use window manager"/>
! </index>
! <pkg path="nfeske/pkg/nano3d/2019-02-22"
! info="simple software-rendering demo"/>
! </index>
Each '<pkg>' node refers to a package with a concrete version and a short
description. By nesting '<index>' nodes, software categories can be defined.
Index files can be published like any other depot content using the
_depot/publish_ tool, which takes care about compressing and digitally signing
the published information:
! ./tool/depot/publish nfeske/index/19.02
To ease the updating of the index with current package versions, the
_sculpt.run_ script creates the _depot/index/<version>_ file from the input
found in _repos/gems/run/sculpt/index_. The latter file is void of any version
numbers.
For any questions about the process, please consult the Genode
[https://genode.org/community/mailing-lists - mailing list].
Showcase of a Java-based network appliance
##########################################
With OpenJDK in good shape (Section [Java]), we have created a Genode scenario
that demonstrates JVM's ability to execute well on embedded hardware. As
target platform, we choose an ARM (i.MX 6) based SoC with two integrated
network interface controllers. In the scenario, the Genode system boots
directly into a Java application, which in turn spawns two HTTP server
instances where each instance communicates through a dedicated NIC. Both
server instances run as one Java program.
[image java_nic_filter]
The Java application (a JAR file) is loaded from the board's SD card, and
therefore, can easily be replaced. If you are interested in Java and ARM SoC,
the full details and a step by step instruction can be found at our
[https://genodians.org/ssumpf/2019-02-27-java-19-02 - Genodians.org] site.
Genodians.org as a showcase of a Genode-based web appliance
###########################################################
[https://genodians.org - Genodians.org] is our take on a federated blogging
platform about Genode-related topics. It does not host the actual content
but rather aggregates content hosted elsewhere.
[image genodians]
The site periodically fetches content in the form of zip archives containing
raw text and PNG images, extracts the archives, and transforms the text into
HTML using a custom static site generator. The result of the transformation
process is served by the lighttpd web server.
This system is based on Genode and can be found here:
:Genodians.org repository:
[https://github.com/genodelabs/genodians.org]
A few noteworthy technical points about the site:
* The periodic process of downloading, extracting, and transforming content
is realized via the 'fetchurl', 'extract', and 'sequence' components.
* The textual content uses the markup of the
[https://github.com/nfeske/gosh - GOSH]
text processing tool, which is similar to
[https://daringfireball.net/projects/markdown/ - Markdown].
* The custom static site generator consists of a plain makefile and a few GOSH
style files. The makefile is executed within a noux instance, which is
Genode's custom Unix runtime environment. It is re-spawned for each
iteration. As GOSH is written in Tcl, the tclsh is used within the noux
environment.
* The lighttpd web server uses a statically supplied SSL certificate.
* The most time-consuming part of creating Genodians.org was the CSS
definition.
* The entire system image is about *8 MiB* in size. It includes the following
ingredients:
* Kernel (e.g., NOVA),
* Basic Genode components such as init, NIC router, the VFS server,
* Network driver (based on iPXE),
* Linux TCP/IP stack as a library,
* Curl, libssh, libssl, libcrypto, lighttpd
(for downloading and serving content),
* libarchive, zlib, liblzma (for extracting the downloaded content)
* Noux, coreutils (stripped down), bash, GNU make, and tclsh (for
transforming text into HTML)
More information about the site's inner working will be posted in a series
of articles - guess where? - at Genodians.org!
Base framework and OS-level infrastructure
##########################################
Removal of deprecated APIs
==========================
Almost three years ago -
with [https://genode.org/documentation/release-notes/16.05 - version 16.05] -
we started the transition to Genode's modern API. One year later - in
[https://genode.org/documentation/release-notes/17.05#Completed_component_transition_to_the_modern_API - version 17.05],
we announced the completion of this transition but we retained the deprecated
APIs to accommodate Genode users that picked up the new API only gradually.
With the current release, we finally drop the deprecated APIs along with a
couple of other legacies:
* The _base/timed_semaphore.h_ has been removed. In hindsight, officially
providing this utility was a big mistake because it lured developers into
a wrong direction. In fact, we found that there is no legitimate use of it
when a component is designed in a clean way. If a component relies on a
timed semaphore, it should better be redesigned. There are two noteworthy
places where a timed semaphore is still used as a band-aid solution:
the 'pthread_cond_wait' implementation of the libc, and DDE for rump
kernels. Those places now host a private copy of the timed semaphore, but
should ultimately be reworked.
* The header _base/printf.h_ has been removed along with the log back end
for 'printf'. The 'Console' with the format-string parser is still there
along with 'snprintf.h' because the latter is still used at a few places,
most prominently the 'Connection' classes.
* The notion of a 'Ram_session' exists no more since the former RAM-session
interface was merged into the PD-session interface in
[https://genode.org/documentation/release-notes/16.05#Consolidation_of_core_s_SIGNAL__CAP__RM__and_PD_services - version 16.05].
Still, the types were preserved (by typedefs to 'Pd_session') to keep up
API compatibility. Those last traces of 'Ram_session' are gone now. The
'Env::ram()' accessor returns the 'Ram_allocator' interface, which is a
subset of the 'Pd_session' interface.
* The use of the global 'Genode::env()' accessor function is not possible
anymore.
* The old 'Child_policy::resolve_session_request' interface that returned
a 'Service' instead of a 'Route' has been removed.
* Boolean accessor methods are no longer prefixed with 'is_'. E.g., instead
of 'is_valid()', use 'valid()'.
* All connection constructors need the 'Env' as argument.
* The 'Reporter' constructor needs an 'Env' argument now because the
reporter creates a report connection.
* The old notion of 'Signal_dispatcher' is gone. For receiving
asynchronous notifications, the 'Signal_handler' interface must be used.
* Transitional headers like _os/server.h_, _cap_session/_,
_volatile_object.h_, _os/attached*_dataspace.h_, and
_signal_rpc_dispatcher.h_ have been removed.
* The distinction between 'Thread_state' and 'Thread_state_base' does
not exist anymore. Only 'Thread_state' prevails.
* The header _cpu_thread/capability.h_ along with the type definition of
'Cpu_thread_capability' has been removed. Use the type
'Thread_capability' defined in cpu_session/cpu_session.h instead.
* The _os/ram_session_guard.h_ has been removed.
Use 'Constrained_ram_allocator' provided by _base/ram_allocator.h_ instead.
Source-tree reorganization
==========================
Timer moved from os to base repository
--------------------------------------
Traditionally, the user-level timer was hosted at the os repository at
_drivers/timer/_. However, since the timer and timeout handling have become
part of the base library (the dynamic linker), the component naturally belongs
to the base repository. It is now located at _base/src/timer/_ and
_base-<kernel>/src/timer/_ respectively.
Note that this change affects include paths for the former
_include/os/timer/_, _include/os/alarm.h_, and _include/os/duration.h_
headers. Those can now be found in _include/base/_.
Consistent naming of block components
-------------------------------------
Regarding the naming of files and APIs, Genode follows the convention of
avoiding abbreviations. Most components follow this convention, with the block
servers being the exception to the rule. Those were named "part_blk" or
"rom_blk". With the current release, we removed this inconsistency by renaming
those offenders, changing "blk" to "block". Closely related, abbreviations
like "cli" and "srv" have been replaced by "client" and "server".
Improved API safety
===================
XML-parsing API
---------------
Genode consistently uses XML for component configurations and for reports
generated by components. The latter are often consumed by other components.
This puts the XML parser into a prominent position. Genode's XML parser
comes in the form of the 'Xml_node' class. Since it was introduced before the
age of modern C++, it offers several risky "C-ish" methods, in particular
accessors that return pointers, raising memory-safety concerns. To promote a
safe programming style, the following parts of the interface are subject to
change now:
* The 'Xml_node::addr', 'Xml_node::content_addr', and 'Xml_node::content_base'
accessors will be removed because it is all too easy to store the returned
pointers and forget about the lifetime of the originating 'Xml_node' object.
Fortunately, in practice, those methods are rarely used because information
is typically represented in attributes, not as node content. However, to
still support the access to the raw content, the new
'Xml_node::with_raw_node', 'Xml_node::with_raw_content', and
'Xml_attribute::with_raw_value' methods call a functor taking the raw byte
buffer and size as arguments. This way, the lifetime of the pointer is
naturally bound to the scope of the functor.
* The new 'with_sub_node' method calls a functor with the specified
sub node as argument and thereby reduces the need for the traditional
'Xml_node::sub_node' method, which returns an 'Xml_node'.
The latter is risky because an 'Xml_node' contains a pointer to the actual
data. In contrast, the lifetime of the 'Xml_node' processed via the new
'Xml_node::with_sub_node' is naturally bound to the scope of the passed
functor.
* The 'Xml_attribute::value' and 'Xml_node::value' methods now take an
argument of type 'T &out' instead of 'T *out', which eliminates the
uncertainty of a possible nullptr argument.
The original interface will still be available for a while but it will
eventually be removed.
Simplified session-policy handling
----------------------------------
Most server components make use of the 'Session_policy' utility for selecting
a client policy depending on the session label. However, the pattern of its
use remained a bit inconsistent across components, in particular the handling
of the case where no policy could be found. Some components outright
denied the session where others implemented a fallback to a built-in default
policy. This inconsistency is now removed.
Following the principle of deny-by-default the absence of a matching policy
denies the session request. Since the 'Session_policy::No_policy_defined'
exception is a typedef to the 'Genode::Service_denied' exception, a server
does not need to explicitly handle it, which simplifies the implementation.
With this change, the 'No_policy_defined' case is always an error case. Hence,
the new version of 'Session_policy' prints a error message, which relieves the
server developer from implementing diagnostics in the server code.
As a consequence of this change, scenarios that used to rely on the
policy-fallback approach of some servers - notably the NIC bridge, window
manager, the window decorators - need a slight adaption: To enable the
fallback to a default policy, a '<default-policy>' node must be explicitly
specified in the server's configuration.
Removed pointers from Genode::Fifo interface
--------------------------------------------
To make the use of the 'Genode::Fifo' data structure more safe, all methods
that return pointers have been replaced by methods that call a functor with
a reference as argument.
New server-side block-request stream API
----------------------------------------
The current block-component API (_os/include/block/_) was designed at a time
long before Genode's modern component API was introduced. Back then, the use
of blocking calls was prevalent. Today, we design components to work
asynchronously. The original block-server API was successively enhanced to
allow the implementation of asynchronous block servers but it remained "upside
down" while growing more complicated than it should be.
To simplify the implementation and verification of block servers, the current
release introduces a modern API that will eventually replace the original
block-component API. The new API is called *block-request stream* and can be
found at
[https://github.com/genodelabs/genode/blob/master/repos/os/include/block/request_stream.h - os/include/block/request_stream.h].
It is designed with the following considerations:
* It anticipates the asynchronous operation of block servers by default.
Using the new API, such servers - in particular block-device drivers - can
be implemented as state machines triggered by client requests and device
interrupts.
* It reinforces the memory safety of the server code by not returning any
pointers or references.
* It relieves the server developers from handling special cases (like a
congested acknowledgement queue) while being flexible enough to accommodate
different categories of components like drivers, resource multiplexers
(part_block), and bump-in-the-wire components in a natural way.
* It naturally supports the batching of requests as well as zero-copy
(device DMA directly into the client's communication buffer).
The use of the new API is illustrated by the artificial test at
[https://github.com/genodelabs/genode/blob/master/repos/os/src/test/block_request_stream/main.cc - os/src/test/block_request_stream/].
We plan to successively migrate all existing block servers to this new API
and will remove the traditional block-component API eventually.
GUI stack
=========
Motivated by our work on Sculpt as described in Section
[Sculpt OS as a community experience (CE)], Genode's GUI stack received
the following improvements:
Window management
~~~~~~~~~~~~~~~~~
To improve the visual appearance of Sculpt's administrative GUI, the *themed*
*decorator* was enhanced with an option to disable decorations based on the
window label. By setting the 'decoration' attribute of a '<policy>' node to
"no", matching windows appear without any border, which is desirable for
Sculpt's component graph.
Furthermore, the themed decorator accepts the '<policy>' attribute
'motion=<number>'. The default value is 0. If a value higher than 0 is
specified, window-geometry changes are applied as an animation where the
<number> denotes the number of animation steps. This feature is used for
smoothing the placement of Sculpt's component graph.
The *motif* *decorator* - which is a nice alternative to the themed decorator -
is now giving visual feedback to mouse clicks. For example, while the user
drags a window by clicking on the window title, the title bar appears as
pressed, which creates an improved sense of responsiveness.
Unified shape-report routing
----------------------------
Applications propagate custom mouse-cursor shapes by issuing "shape" reports
towards the pointer component. The pointer component correlates the session
labels of the incoming shape reports with the label of nitpicker's currently
hovered client. Hence, an application's shape report session should take the
same route as its nitpicker session.
The presence of the window manager as an intermediary of the nitpicker session
breaks this rule. Consequently, rather awkward label-rewriting magic was
required when routing shape reports originating from windowed applications. To
make the shape reporting more natural, the window manager has become able to
proxy shape reports on behalf of its clients. When routing a shape through the
window manager, the labels of both the nitpicker sessions as well as the shape
report sessions contain the intermediary "wm ->" part.
Unicode support for the graphical terminal
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The terminal has been changed to represent characters as 16-bit codepoints
internally. It thereby became able to display a much larger variety of
international characters when using a suitable font. Unicodes are sent through
the terminal session via bursts of UTF-8 bytes now.
As a collateral change, the generic 'Codepoint' class became printable via
Genode's 'Output' interface. This way, a codepoint can easily be serialized
into UTF-8 bytes.
Programming languages
#####################
Ada and SPARK
=============
_The work described within this section was contributed by_
_[https://componolit.com - Componolit]. Thanks to Alexander Senier_
_and Johannes Kliemann for the fantastic collaboration!_
The integration of SPARK/Ada programs was improved greatly and the Ada runtime
was renamed to 'spark'. This emphasizes its main purpose: trusted components
that can be formally verified. A more feature-rich runtime based on 'libc' was
added to genode-world under the name 'ada'.
Build system integration
------------------------
Up until now, the (deprecated) 'gnatmake' tool was used to build Ada object
files. This was unfavorable for two reasons: First, multiple invocations of
gnatmake would sporadically corrupt the compiler-generated Ada linker files
('.ali') and break parallel builds. Second, Ada dependency information did not
get propagated into Genode's build system such that certain source changes
failed to trigger a rebuild.
For these reasons the 'gnatmake' tool has been dropped in favor of regular
compiler calls as done for all other languages. To facilitate consistent
rebuilds on source-code changes, the
[https://github.com/Componolit/ali2dep - ali2dep] tool was created. From an
'.ali' file, it produces '.d' files suitable for direct inclusion into
Genode's build system.
Until the integration into the toolchain, 'ali2dep' support needs to be
enabled through the 'CUSTOM_ALI2DEP' variable (absolute path or command name
if the command is in 'PATH'). By default, a warning about the absence of the
tool is emitted and dependency information is not generated. You can add this
variable to the 'etc/tools.conf' of your build directory as follows:
! CUSTOM_ALI2DEP = /path/to/ali2dep
Elaboration code
----------------
Previously, Ada programs that required elaboration code to be run were
unsupported on Genode. With this release, Ada programs are bound using
'gnatbind', which results in elaboration code being generated. An additional
benefit comes in the form of proper error messages at compile time if, for
example, source code is missing or outdated.
To run an Ada main program with elaboration, calls to 'adainit()' and
'adafinal()', generated by the binder, need to be added to your
component-construction code:
! extern "C" void _ada_main(void);
! extern "C" void adainit();
! extern "C" void adafinal();
!
! void Component::construct(Genode::Env &env)
! {
! adainit();
! _ada_main();
! adafinal();
! env.parent().exit(0);
! }
Note, that the name of the Ada main program ('_ada_main()') in this example
depends on the name of your main procedure.
Debug output
------------
Support for 'GNAT.IO' was added to the runtime. 'GNAT.IO' is a stripped-down
text I/O facility, which we map to a terminal session on Genode. Only output
is supported at the moment. A pointer to a terminal session has to be provided
to the ADA runtime in order to use 'GNAT.IO'. Please refer to
_repos/libports/src/test/gnatio/_ as an example.
Unit testing
------------
To facilitate test-driven development of Ada components, the
[https://www.adacore.com/documentation/aunit-cookbook - AUnit] unit testing
framework was ported to Genode. It is located in the genode-world repository
whereas a usage example can be found at 'src/test/aunit'. Note that the full
Ada runtime ('ada') is required as the framework uses features not present in
the SPARK runtime.
Java
====
Since Genode release
[https://genode.org/documentation/release-notes/18.11#Java_language_runtime - 18.11],
we continued our effort to enable the just-in-time (JIT) compiler for OpenJDK.
We are happy to announce that we were able to achieve this goal. The JIT
compiler is now enabled as default for both x86 (64 bit) and ARM (32 bit),
which significantly improves the performance of the Java virtual machine for
these architectures.
Additionally, we even further improved JVM's performance by taking advantage
of more aggressive compiler optimizations. This triggered some unidentified
bugs and led to a greatly enhanced stability of OpenJDK.
For a small hello world example we offer a simple run script:
! make run/java
For a more complex scenario please refer to Section
[Showcase of a Java-based network appliance].
Nim
===
Support for Nim has been removed from the Genode build system to encourage
building Nim components out-of-tree using the Nimble package manager.
The compatibility of the base system with the Nim runtime will be monitored
and improved regardless of this change.
OCaml
=====
A proof-of-concept port of the OCaml bytecode interpreter has been placed
in the world repository. This allows simple programs that are compiled to the
bytecode instructions to be executed and is a first step in supporting the
OCaml language. This interpreter does not yet include the standard library.
Therefore only language primitives are available. Porting the standard library
appears to simply be a matter of defining a build process. The greatest
hurdle to porting existing OCaml applications would seem to be finding a path
into the workflow of the OPAM package tooling.
Libraries and applications
##########################
New utility for taking screenshots
==================================
The flif_caputure screenshot utility has been added to the world repository.
This utility implements a framebuffer and input service which it proxies to
its parent. When the utility observes the *PrtSc* key, it captures the content
of the framebuffer and writes it to the file system in the FLIF image format,
FLIF being chosen primarily for its uncomplicated API. To use the utility in
practice, the common case would be to run it as a client of the default window
manager with a second window manager stack running as a client of the capture
utility. This allows capture behavior and scope to remain simple and explicit.
The background story behind the tool is covered by a dedicated
[https://genodians.org/ehmry/2019-01-31-flif_capture - posting] at Genodians.org.
Growing use of the Genode-World repository
==========================================
The [https://github.com/genodelabs/genode-world - Genode-World] repository
contains software that is not strictly part of the official Genode OS
framework but rather supplemental. In particular, it contains ports of
3rd-party software to Genode. The pool of ported software is steadily growing,
which moves the world repository more and more into the spotlight of Genode
users. For example, among a variety of games and experimental components, our
port of OpenJDK ([Java]) is also hosted there. To acknowledge the growing
importance of the world repository, we added the building of all world depot
archives to our nightly build tests.
With this baseline of quality assurance in place, it was a good time to move
supplemental software such as Dosbox, FUSE, libav, libSDL, and its companion
libraries from the Genode repository to the Genode-World repository. Speaking
of libSDL, as part of the curation of the world content, the library back end
of libSDL has been changed to the direct use of the nitpicker session
interface instead of the lower-level framebuffer and input interfaces. Thanks
to this change, many scenarios could be greatly simplified, in particular
packages designated for the use in Sculpt OS.
Updated or removed 3rd-party software
=====================================
The following 3rd-party software received an update:
* OpenSSL updated to version 1.0.2q with 'SSL_CONF_*' enabled,
as needed by lighttpd's mod_openssl.
* Lighttpd updated to version 1.4.52, with TLS enabled.
* libpng updated to version 1.6.36
* jbig2dec updated to version 0.15
Removal of VirtualBox 4
-----------------------
All regular users of Genode's VirtualBox port migrated to version 5 a long
time ago. Version 4 was still maintained because this is the only version
supported on the Muen separation kernel. However, since the focus of the Muen
developers recently moved towards the use of nested virtualization, our
version of VirtualBox 4 is no longer vital for Muen. So we could remove it.
Platforms
#########
Board support for i.MX6 Quad Sabrelite and Nitrogen6 SoloX
==========================================================
In 2018, we extended our device-driver support for NXP i.MX 5 and 6 based ARM
platforms. This year, we continue our commitment to this platform with
additional support for the NXP reference board i.MX6 Quad Sabrelite and the
Nitrogen6 SoloX, which features dual Gigabit Ethernet.
Resizeable virtual framebuffer on Linux
=======================================
When executing Genode on Linux, we use a libSDL-based pseudo driver for
running graphical scenarios. This *fb_sdl* server has now been enhanced with
the ability to resize the SDL window. Such a resize event is translated into
Genode's framebuffer resize protocol. Thereby, the resizing of the host window
looks like a mode change to Genode's GUI stack. This greatly eases the testing
of the mode-change handling of components like the nitpicker GUI server.
Tooling and build system
########################
Enforced 'override' annotations
-------------------------------
The '-Wsuggest-override' warning complains about implementations of virtual
functions that lack the override keyword. If the implementation of a virtual
method is marked as 'override' the compiler checks for a matching virtual
method in the base class. If there is no such method, the implementation
unexpectedly diverged from the interface.
An interface may change over time, which is especially troublesome when it
contains a default implementation of a virtual method. Without 'override'
annotations, the compiler will silently add the outdated implementation of the
derived class as an overload of the interface's default implementation, which
introduces a subtle but potentially serious bug.
To rule out such bugs in the future, we made 'override' annotations mandatory
when using the default strict warning level.
Generalized use of the depot by run scripts
-------------------------------------------
With more and more run scripts using archives out of Genode's depot, we gained
experience with typical usage patterns. In particular, we found the universal
use of the configurable depot user preferable over the hard-wiring of
'genodelabs'. Hence the 'depot_user' function has become a built-in function
of the run tool.
Repeat mode for the depot-autopilot test environment
----------------------------------------------------
The _depot_autopilot.run_ script now supports the environment variable
'TEST_REPEAT' with the possible values 'until_forever' and 'until_failed'.
When specified, the tests are run in a loop. The latter argument is
particularly useful for reproducing sporadic errors.
New run script to execute a single test w/o the depot
-----------------------------------------------------
The new _os/run/test.run_ script allows for the quick execution of an
individual test from the build directory while side-stepping the depot.
It expects a 'PKG' variable specifying the test package. E.g.,
! make run/test KERNEL=nova PKG=test-xml_node
Note that it does not cover all test packages right now.
The tool and its limitations are explained in more detail by a dedicated
[https://genodians.org/nfeske/2019-02-05-shortcut-for-testing - posting]
at Genodians.org.
Support for undefined-behavior sanitizer
----------------------------------------
The
[https://developers.redhat.com/blog/2014/10/16/gcc-undefined-behavior-sanitizer-ubsan/ - UndefinedBehaviorSanitizer]
(UBSan), also described in the
[https://gcc.gnu.org/onlinedocs/gcc/Instrumentation-Options.html - GCC documentation],
can detect undefined behavior while a program is running.
With the current release, this diagnostic feature of GCC becomes available for
analyzing Genode components. By specifying 'SANITIZE_UNDEFINED = yes' in a
'target.mk' file, the '-fsanitize=undefined' compiler flag is enabled and the
program is linked with libubsan and libsanitizer_common. The program has to
call 'env.exec_static_constructors()' and 'sanitizer_init(env)' upon startup
to initialize the sanitizer libraries. Whenever undefined behavior is detected
while the program is running, a "runtime error:" message including the source
code location of the error is printed to the log.

886
doc/release_notes/19-05.txt Normal file
View File

@@ -0,0 +1,886 @@
===============================================
Release notes for the Genode OS Framework 19.05
===============================================
Genode Labs
The Genode release 19.05 is primarily focused on platform support.
It adds compatibility with the 64-bit ARM architecture (AARCH64),
comes with improvements of the various kernels targeted by the framework,
and extends the list of supported hardware. The increased diversity of base
platforms calls for unifications to keep the hardware and kernel landscape
manageable.
On that account, Genode uses one reference tool chain across all kernels
and CPU architectures. The current release upgrades this tool chain to
*GCC 8.3* with C++17 enabled by default
(Section [Tool chain based on GCC 8.3.0 and binutils 2.32]).
To increase the velocity of Genode system scenarios across different boards
of a given CPU architecture, the release introduces the notion
of *board and kernel-agnostic build directories* presented in Section
[Unified build directories for ARM]. Once built for one particular
CPU architecture, the same binaries can be deployed at any supported board or
kernel without recompilation. This vastly accelerates the workflow when
targeting multiple boards and emulators at once.
As another major unification effort, the current release introduces a new
*kernel-agnostic virtualization* interface. Up until now, virtualization
used to be inherently tied to a specific kernel. Thanks to the new interface,
however, one virtual machine monitor implementation can be combined with
kernels as different as NOVA, seL4, or Fiasco.OC. No recompilation needed.
As outlined in Section [Kernel-agnostic virtual-machine monitors], Genode
has now become able to run the Seoul VMM on all those kernels, while
VirtualBox is planned to follow.
On our [https://genode.org/about/road-map - road map], we originally
planned several user-facing features related to Sculpt OS. However, in the
light of the major platform efforts, we decided to defer those topics instead
of rushing them.
That said, the release is not without new features. For example, our port
of *OpenJDK* has become able to host the Spring framework and the Tomcat web
server, there are welcome improvements of the *package-management tooling*,
and we added new options for user-level networking.
Finally, version 19.05 is accompanied with the annual revision of the *Genode*
*Foundations book* (Section [New revision of the Genode Foundations book]),
which is now available as an online version in addition to the regular PDF
document.
Kernel-agnostic virtual-machine monitors
########################################
Since the introduction of Genode's
[https://genode.org/documentation/release-notes/17.02#Genode_Application_Binary_Interface - Application Binary Interface]
in the 17.02 release,
Genode components can be assembled once for a given hardware platform and
executed without further adjustments on all the supported kernels. However, at
that time, the supported virtual machine monitors - a port of VirtualBox 4 & 5,
Seoul, and our
[https://genode.org/documentation/articles/arm_virtualization - custom VMM] -
remained kernel specific.
Of course, last remaining bastions tempt to be taken! So last year, we started
the venture to unify our virtualization interface across different kernels.
Starting point was the already existing Genode VM interface of our custom VMM
on ARM. We took it and extended the interface with caution to the x86 world.
Having an eye on the requirements of our already supported VMMs on NOVA(x86),
namely VirtualBox and Seoul, the VM interface got extended with missing
features like multiple vCPU support and specific VM handlers per vCPU.
In parallel, we started to investigate the other x86 microkernels with regard
to hardware-assisted virtualization features, namely seL4 and Fiasco.OC.
Over several weeks, we iteratively extended the interface. On the one hand
we familiarized ourself with the kernel interfaces of seL4 & Fiasco.OC while
on the other hand considered known requirements of the NOVA microhypervisor.
Additionally, we kept our custom VMM for ARM still compatible with the new VM
interface.
During this time, it became apparent that the control flow on a VM resume/pause
and a VM event(exit) are different between seL4/Fiasco.OC and NOVA/base-hw.
For seL4 and Fiasco.OC, a VM is resumed by making a blocking syscall on the
kernel. On a VM event, the blocking syscall would return. Logically, on both
kernels the VMM 'calls' into the VM.
On base-hw and NOVA, it is the other way around. Whenever a VM causes a VM
event, the kernels set up either an asynchronous notification (base-hw) or a
synchronous IPC call (NOVA) to the VMM. In both cases the VMM executes a prior
registered VM event handler as response.
Upon return of the VM event handler, the kernel resumes the VM. Logically, on
NOVA and base-hw the VM 'calls' into the VMM. The following two figures
contrast the different flows of control between a user-level virtual machine
monitor and the respective kernels.
[image vm_seq_foc_sel4]
Control flow of handling virtualization events on Fiasco.OC and seL4
[image vm_seq_nova_hw]
Control flow of handling virtualization events on NOVA and the base-hw kernel
Hiding this differences behind a common VM interface was the challenge we were
faced, accepted, and won. Finally, at one point in December we had all 3
x86 kernels running with a test VMM - without re-compilation. The toy VMM
(vmm_x86.run) runs multiple vCPUs on multiple physical CPUs and tests several
VM events/exits.
After this major breakthrough, we spent the days left before Christmas to
adjust the Seoul VMM to the new VM interface, freeing it from the ties to the
NOVA kernel. The choice to start with Seoul stems from the fact that it is -
compared to VirtualBox - much smaller and therefore easier to debug if things
go wrong in the beginning. After one week, the Seoul VMM became in principle
kernel independent and worked again on NOVA. After some more days, it started
to hobble on seL4 and Fiasco.OC as well.
With the New Year, VirtualBox was the next target where all NOVA kernel
specific calls were replaced with the new Genode VM interface. Mid of January,
the work showed first results by having a prototype running simple VMs on NOVA
again. At this point, it became apparent that this venture is not anymore an
adventure. All the findings and technical details so far got condensed to a
[https://fosdem.org/2019/schedule/event/microkernel_virtualization - presentation]
given and recorded at the [https://fosdem.org/2019 - FOSDEM 2019] in Brussels
in February in the
[https://fosdem.org/2019/schedule/track/microkernels_and_component_based_os - Microkernel and Component based OS]
developer room.
At this point, we started transforming our prototype for the 4 kernels into a
clean solution to be featured in Genode 19.05. Eventually, the kernel-agnostic
Seoul VMM runnable on seL4, Fiasco.OC, and NOVA entered Genode master. In the
Genodians article
[https://genodians.org/alex-ab/2019-05-09-seoul-vmm - Seoul VMM and the new VM interface],
we conserved the current state and a few performance measurements.
Shortly before this release, the kernel-agnostic VirtualBox VMM version on
Genode/NOVA got ready. The kernel-agnostic version is in principle capable to
run Linux VMs and Windows 7/10 VMs on Genode/NOVA. Currently, this version
must still be considered as experimental and does not run on seL4 or
Fiasco.OC.
Because of the experimental nature of the kernel-agnostic VirtualBox VMM
version, we decided to keep the kernel-specific version for NOVA for the
moment. This gives us time to test and improve the kernel-agnostic version. It
also allows us to compare both versions to each other.
If time and interest permits, we will consider bringing the virtualization
support on Genode/seL4 and Genode/Fiasco.OC on par with Genode/NOVA.
When building VirtualBox with Genode 19.05,
you will find both the 'virtualbox5-nova' and the new 'virtualbox5' binaries
in the build directory. The former relies on NOVA's kernel interface whereas
the latter uses Genode's kernel-agnostic VM interface. Nightly tested run
scenarios with the new VM interface are named 'vbox5_vm*.run' and can be found
in the 'repos/ports/run' directory.
Broadened CPU architecture support and updated tool chain
#########################################################
With the major update of Genode's tool chain and library infrastructure in
tandem, the framework gains a consistent architecture support across x86-32,
x86-64, ARM-32, RISC-V, and the newly added AARCH64. This includes the tool
chain (Section [Tool chain based on GCC 8.3.0 and binutils 2.32]), the base
framework, the dynamic linker, and the C runtime
(Section [Updated dynamic linker and C runtime]).
Together with this update, we took the chance to wrap up our long-time move
away from board-specific build directories to one generic build directory
shared by multiple kernels and boards for a given CPU architecture
(Section [Unified build directories for ARM]).
Tool chain based on GCC 8.3.0 and binutils 2.32
===============================================
Genode uses a tailored tool chain based on GCC and binutils that is used
across all supported kernels and architectures. Since the previous tool-chain
update in version
[https://genode.org/documentation/release-notes/17.05#Tool_chain - 17.05],
we relied on GCC 6.3. After two years, it was time for an update, motivated by
three major reasons. First, the C++17 standard is common-place now. We Genode
developers anticipate the improvements that come with it. Second, RISC-V and
AARCH64 are now supported by mainline GCC. Up till now, we had to maintain a
custom patch set for Genode's RISC-V support. AARCH64 was not supported yet.
Third, our increasing engagement with SPARK depends on recent improvements of
the Ada compiler that is part of GCC.
With Genode 19.05, the tool chain is now based on binutils version 2.32, GCC
version 8.3.0, GDB version 8.2.1, gcov version 8.3.0, standard C++ library
version 8.3.0.
The tool chain supports x86 (32 and 64 bit), ARM, AARCH64, and RISC-V.
For C++ code, the C++17 standard is enabled by default.
The update of the tool chain provided a perfect opportunity to replace the
former use of gnatmake with a much more natural integration of Ada in Genode's
build system, using a custom ali2dep dependency-extraction tool developed
by [https://github.com/Componolit/ali2dep - Componolit].
In contrast to the previous versions, we switched to a versioned installation
directory for the new tool chain. By default, it is now installed to
_/usr/local/genode/tool/19.05/_. This eases the use of different tool-chain
versions for different development branches.
:Tool-chain installation:
[https://genode.org/download/tool-chain]
Caveats
-------
The tool-chain update required a number of adaptations throughout the source
tree, and may affect Genode users too:
* The silent fall-though within switch statements must now be replaced
by an explicit annotation of the form
! [[fallthrough]]
* The 'register' keyword is no longer valid with C++17. Hence, it must
be removed from the code.
* Types marked as 'Noncopyable' can no longer have an implicit default
constructor. A default constructor must be provided manually.
Updated dynamic linker and C runtime
====================================
The tool-chain update is accompanied with a major update of the dynamic linker
and the C runtime to cover both the AARCH64 and RISC-V architectures in
addition to the traditional x86 and ARM architectures.
FreeBSD 12 supports AARCH64 and RISC-V. Hence, by updating our C runtime to
this version, Genode's libc support extends to those architectures now.
Until now, Genode's dynamic linker supported only the eager binding of symbols
at loading time on the *RISC-V* architecture. With the current version, we
lifted this limitation in favor of lazy binding as used on all other CPU
architectures.
Unified build directories for ARM
=================================
In version
[https://genode.org/documentation/release-notes/17.02#Genode_Application_Binary_Interface - 17.02],
we introduced unified build directories for x86, which allow us to build and
run Genode scenarios on various kernels while using only one build directory.
This concept leverages Genode's cross-kernel binary compatibility to make
the switch from one kernel to another - like developing on base-linux and
deploying on base-nova - a seamless experience.
On ARM, this concept was held back by a third dimension. The
system-integration step does not only depend on the CPU architecture and
the kernel but also on the used board. Our traditional approach was the
use of one build directory per board. Granted, within such a build directory,
one could easily switch between different kernels like Fiasco.OC and seL4.
But on ARM, we find an extreme proliferation of different board
configurations, which share the same CPU architecture but demand different
integration steps. This ensues large redundancies among different build
directories. Switching from one board to another - even when most binaries
happen to be exactly the same - requires an additional rebuilding effort.
With version 19.05, we took the chance to generalize the unified build
directory concept to support multiple different boards per build directory,
greatly reducing the friction when switching kernels and boards for a given
CPU architecture (like ARMv7a). This change has the following implications:
* Drivers no longer depend on the SPEC values as configured for a build
directory.
* All *binaries* are now *named unambiguously*. For example, the USB drivers
for the Panda (OMAP) and Arndale (Exynos) boards were formerly called
'usb_drv' but were different programs. They just never happened to
appear in the same build directory. In the new version, they are named
'panda_usb_drv' and 'arndale_usb_drv' respectively and can thereby
peacefully co-exist within the same 'armv7a' build directory.
Note that this binary renaming will likely affect existing run scripts.
* Include paths no longer hide the board details, which makes the included
code much more easy to follow.
* Run scripts need to pick the right binary, depending on the used board.
Since the board is no longer tied to a build directory, the selection
of the used board has become a build-time variable 'BOARD' following
the successful pattern of how we specify the targeted 'KERNEL'.
To avoid the pollution of run scripts with difficult conditions, we wrap
the drivers needed for a particular board and use case into so-called
_drivers_ packages. Such a package can be instantiated within a generic
scenario using a nested init instance. The details about the drivers and
how they access the hardware remain nicely hidden inside this building block.
Currently, there exist _drivers_ packages for two distinct use cases:
:drivers_interactive pkgs: contain all drivers needed for simple
interactive scenarios, including graphical output and user input.
:drivers_nic pkgs: contain the drivers needed for communication over the
network.
Whenever a run script fits one of these use cases, it can rely on the
corresponding ready-to-use drivers packages via:
! import_from_depot [depot_user]/src/[base_src] \
! [depot_user]/pkg/[drivers_nic_pkg] \
! ...
With the drivers package incorporated, the drivers subsystem can be
instantiated as follows (note the absence of any board or kernel-specific
details):
! <start name="drivers" caps="1000">
! <resource name="RAM" quantum="32M" constrain_phys="yes"/>
! <binary name="init"/>
! <route>
! <service name="ROM" label="config">
! <parent label="drivers.config"/> </service>
! <service name="Timer"> <child name="timer"/> </service>
! <any-service> <parent/> </any-service>
! </route>
! <provides> <service name="Nic"/> </provides>
! </start>
Using the 'BOARD' build variable
--------------------------------
The new 'BOARD' variable selects the board to use. It can be specified either
as a 'make' command-line argument (or environment variable), or defined in the
build-directory configuration (_etc/build.conf_). The following boards are
available:
:arm_v6: rpi
:arm_v7a: arndale, imx53_qsb, imx53_qsb_tz, imx6q_sabrelite, imx7d_sabre,
nit6_solox, odroid_x2, odroid_xu, panda, pbxa9, usb_armory,
wand_quad, zynq_qemu
:arm_v8a: rpi3
:x86_64: pc, linux, muen
:x86_32: pc, linux
:riscv: spike
Please note, when running Genode on Linux or the Muen separation kernel -
although it is run on common x86 PC hardware - we treat both runtime
environments as separate "boards" because their device driver environments
are fundamentally different.
New revision of the Genode Foundations book
###########################################
The "Genode Foundations" book received its annual update, which is actually
rather a refinement than a revision. The noteworthy additions and changes are:
: <div class="visualClear"><!-- --></div>
: <p>
: <div style="clear: both; float: left; margin-right:20px;">
: <a class="internal-link" href="https://genode.org">
: <img class="image-inline" src="https://genode.org/documentation/genode-foundations-title.png">
: </a>
: </div>
: </p>
* Component health monitoring
* Static code analysis
* Documentation of --depot-user and --depot-auto-update
* Minor adjustments in the under-the-hood chapter
* Changes of the build system
* Updated tool requirements
* Updated API reference
: <div class="visualClear"><!-- --></div>
To examine the changes in detail, please refer to the book's
[https://github.com/nfeske/genode-manual/commits/master - revision history].
New online version of the book
------------------------------
We are happy to announce that the Genode Foundations book is now available
as an online version in addition to the regular PDF version.
:Browse the Genode Foundations book online:
[https://genode.org/documentation/genode-foundations/19.05/index.html]
Thanks a lot to Edgard Schmidt for creating the tooling for the HTML version
of the book!
Base framework and OS-level infrastructure
##########################################
Modernized block-storage interfaces
===================================
With the current release, we revisited Genode's interfaces for accessing
block devices to ease the implementation of asynchronous I/O, to accommodate
zero-copy block drivers, and to support trim and sync operations.
Revised RPC interface
---------------------
The 'Block::Session' RPC interface remained untouched for a long time.
We have now rectified long-standing deficiencies.
First, *sync requests* used to be handled as synchronous RPCs. This is bad
for components like part_block that multiplex one block device for multiple
clients. One long-taking sync request of one client could stall the I/O for
all other clients. The new version handles sync requests as asynchronous
block-request packets instead.
Second, the new version allows a server to dictate the *alignment* of
block-request payload. This way, a driver becomes able to use the payload
buffer shared between client and server directly for DMA transfers while
respecting the device's buffer-alignment constraints.
Third, we added support for *trim* as an asynchronous block operation.
However, as of now, this operation is ignored by all servers.
Fourth, each block operation can now be accompanied with a client-defined
request tag independent from the other parameters of the operation. The tag
allows a block-session client to uniquely correlate acknowledgments with
outstanding requests. Until now, this was possible for read and write
operations by taking the value of the request's packet-stream offset. However,
sync and trim requests do not carry any packet-stream payload and thereby lack
meaningful and unique offset values. By introducing the notion of a tag, we
can support multiple outstanding requests of any type and don't need to
overload the meaning of the offset value.
New client-side API
-------------------
We have now equipped the 'Block::Connection' with a framework API for the
implementation of robust block-session clients that perform block I/O in an
asynchronous fashion.
An application-defined 'JOB' type, inherited from 'Connection::Job',
encapsulates the application's context information associated with a block
operation.
The life cycle of the jobs is implemented by the 'Connection' and driven by
the application's invocation of 'Connection::update_jobs'. The 'update_jobs'
mechanism takes three hook functions as arguments, which implement the
applications-defined policy for producing and consuming data, and for the
completion of jobs.
We plan to gradually move the existing block clients to the new API to benefit
from the latency-hiding effects of asynchronous I/O. The first updated client
is the _block_tester_ component located at _os/src/app/block_tester/_, which
received a number of new features like the choice of the batch size. Please
refer to the accompanied README for a detailed description of the
block-tester.
Unified types for time values
=============================
[https://genode.org/documentation/release-notes/17.05#New_API_for_user-level_timing - Two years ago],
we introduced the so-called timeout framework to provide a general solution
for requirements unmet by the bare timer-session interface - most notably
timer-session multiplexing amongst multiple timeouts, and microseconds
accuracy. Up to this day, the timeout framework has proved itself many times
in both real-life appliances and artificial tests and has become the standard
front end for timing in Genode applications.
With this release, we solved one of the few remaining limitations with the
framework by enabling timeouts of up to 2^64 microseconds (> 500000 years)
across all supported architectures. In order to achieve this, we replaced the
former machine-word-wide types used for plain time values by unsigned 64-bit
integers. We did this not only inside the timeout framework but also to almost
all code in the basic Genode repositories that uses the framework.
By doing so, we also paved the way for a second step, in which we are planning
to replace plain time values as far as possible with the abstract 'Duration'
type. With this type in place, the user wouldn't have to worry anymore about
any plain-integer implications when calculating with time values.
Support for chained EBR partitions
==================================
Having an active community around Sculpt leads to bugfixes in unexpected
places. By now we prefer to use a GPT rather than an MBR based partition table
and although we test 'part_block', the component that parses the tables, on
regular basis, the handling of chained EBR's was flawed. Community member
[https://genodians.org/valerius/index - Valery Sedletski] who relies on such a
setup encountered this flaw and provided a bug report, which enabled us to
quickly reproduce and fix the problem.
IP forwarding with port redirection
===================================
The NIC router can now be used to redirect to individual destination ports on
port-forwarding. To express the redirection, the new 'to_port' attribute can
be added to '<tcp-forward>' and '<udp-forward>' rules in the NIC router
configuration. If the new attribute isn't added, the rules behave as usual and
forward with an unaltered destination port.
Libraries, languages, and applications
######################################
Ada/SPARK runtime and SPARK-based cryptography
==============================================
The SPARK runtime has been updated to GCC 8.3. SPARK components do not require
'Genode::Env' or a terminal session anymore. Debug messages can still be
printed using 'GNAT.IO', which uses 'Genode::log' and 'Genode::error'
internally now.
Threading support, which was never fully implemented, has been removed to
further simplify the runtime. This simplification allowed us to prove absence
of runtime errors for the secondary stack allocator and other parts of the
runtime.
[https://github.com/Componolit/libsparkcrypto.git - Libsparkcrypto] is a
library of common cryptographic algorithms implemented in SPARK. It is
free-standing and has a very small footprint. The port of libsparkcrypto for
Genode has been added to the libports repository. Thanks to Alexander Senier
and Johannes Kliemann of [https://componolit.com - Componolit] for maintaining
the Ada/SPARK runtime and libsparkcrypto.
To accommodate the use case of block encryption, we added the small wrapper
library 'aes_cbc_4k' around libsparkcrypto that provides a simple C++
interface for the en/decryption of 4 KiB data blocks. It uses AES-CBC while
incorporating the block number and the private key as salt values.
Improved resilience of the sequence tool
========================================
We have a simple component that starts other components sequentially. It
will exit whenever one of those components has exited with an error.
However, this component is used by our [https://genodians.org - Genodians]
appliance where it controls the content-update mechanism. Since updating
involves fetching content via HTTP/S depending on external events, e.g.,
the remote site is not reachable, the sequence tool might exit. In a long
running appliance, this is obviously not a useful action where no one is
in place to restart the sequence tool. Rather than increasing the overall
complexity of the appliance by introducing such a management component, we
added a _keep-going_ feature to the sequence tool that will instruct it
to carry on even if one of the started components has failed.
Please look at _repos/os/src/app/sequence/README_ for instructions on
how to use the feature.
NIC-bus server for private LANs
===============================
The 'nic_bus' server was added to the world repository as an alternative
to the 'nic_router' and 'nic_bridge' components. The name may be a slight
misnomer, but this component acts neither as a hub, switch, or router.
The 'nic_bus' implements unicast and multicast Ethernet packet switching
between sessions, but drops any unicast packet not destined for a session
attached to the bus. This is in opposition to the behavior of a typical
Ethernet switch and is intending to create simple, software-defined
local-area-networks for native components as well as virtual machines.
In practice the component has been used for attaching VMs to the
[https://yggdrasil-network.github.io/ - Yggdrasil] overlay network via
a bus-local IPv6 prefix.
Distributed Genode
==================
In
[https://genode.org/documentation/release-notes/16.08#Network-transparent_ROM_sessions_to_a_remote_Genode_system - 16.08],
we initially released the _remote_rom_ components that act as communication
proxies. A communication proxy transparently relays a particular service to
another Genode system. As the name suggests, the remote_rom relays ROM
sessions.
Originally implemented as a proof of concept using bare IP packets, broadcast
MACs and static configuration of IP addresses, we added several improvements
to allow a more general use. First, we adopted the size-guard idea for packet
construction and processing from the NIC router. Furthermore, we adopted the
single-threaded implementation style that was already established in other NIC
components. Thanks to Edgard Schmidt for this contribution. Second, we
implemented ARP requests to eliminate broadcasting. Third, we moved from bare
IP packets to UDP/IP and implemented a go-back-N ARQ strategy in order to
reliably transmit larger ROM dataspaces.
As the remote_rom proved valuable for distributing functionality across
multiple Genode devices, we also applied this concept to the LOG session in
order to transmit LOG output from a headless Genode device to a
[https://genode.org/download/sculpt - Sculpt] system for instance. The udp_log
component provides a LOG service and sends the LOG messages as UDP packets to
another machine. The log_udp reverses this process by receiving these UDP
packets and forwarding the messages to a LOG service. An example can be found
in the world repository at _run/udp_log.run_ and _run/log_udp.run_.
Seoul and VirtualBox virtual machine monitors
=============================================
Besides the conversion of the Genode back end of Seoul to the new VM
interface, we added mouse-wheel support to the PS/2 model and changed the VMM
to request a single GUI/nitpicker session rather than distinct framebufer and
input sessions.
Similar to the Seoul VMM, the VirtualBox VMM was adjusted to the new VM
interface and now uses the GUI/nitpicker session. The original kernel-specific
VirtualBox version tied to the NOVA kernel is still available. Both versions
can be used simultaneously.
Use of Nim decoupled from Genode build system
=============================================
With this release, all integration with Nim tooling has been removed from the
Genode build system as a result of maturing support for additional languages
via Genode SDKs. Building Nim components independently of the Genode source
tree has the benefit of smaller upstream checkouts and faster build times, and
has yielded components such as the
[https://genodians.org/ehmry/2019-03-22-depot_9P - 9P server] used in some
Sculpt developer workflows. An example of an independent build system for Nim
components is
[https://genodians.org/ehmry/2019-04-27-nim_packaging - documented on the Genodians blog].
OpenJDK improvements
====================
Within the 19.05 release cycle, we further improved Genode's OpenJDK support
by enabling additional networking infrastructure required by the
[https://spring.io - Spring Framework]. The improvements especially concern
support for SSL connections, which enabled us to successfully execute an embedded
[https://docs.spring.io/spring-boot/docs/current/reference/html/howto-embedded-web-servers.html - Tomcat]
server natively on Genode x86 and ARMv7 platforms using the same JAR archive.
This line of work continues our Java for embedded systems effort as described in
our [https://genodians.org/ssumpf/2019-02-27-java-19-02 - Boot2Java] article.
Having these features in place, our Java efforts will continue in the direction
of Java Swing and the support of input devices in the future, with the ultimate
goal of seamless Java application integration into
[https://genode.org/download/sculpt - Sculpt OS].
Device drivers
##############
Improved Zynq board support
===========================
The initial support of the Xilinx Zynq-7000 SoC was added to our custom kernel
in 15.11. Since then, the support of this hardware has been incrementally
extended. The definitions of memory maps, frequencies, and RAM sizes for
different Zynq-based boards are found in the world repository.
One of the major additions in this release is the initialization of the L2
cache. In this context, we also added a simple cache benchmark at
_repos/os/run/cache.run_ that measures the access times for memory regions of
different size and thereby reveals the number of cache levels and their sizes.
With the latest improvements of the network driver in 18.11, a zero-copy
approach was introduced as an effort to eliminate bottlenecks in the driver's
performance. However, this modification also introduced a kernel dependency of
the driver in order to flush packet-buffer memory from the cache before handing
it over to the DMA-controller. With this release, we moved back to using
uncached dataspaces in order to eliminate the cache flushes and the kernel
dependency. Interestingly, we could not recognize a significant impact on the
driver's performance, which confirms the presumption that flushing the cache
nullifies the gain from using cached dataspaces.
In order to enable the continuous operation of the network driver, we extended
the driver-internal error handling that is necessary to recover the network
driver in certain situations.
_Thanks to Johannes Schlatow for contributing and maintaining Genode's Zynq support!_
Updated Intel network drivers
=============================
As a result of recurring issues with modern Intel i219 laptop NICs, we
updated the driver sources for Intel chipsets to the latest upstream
iPXE version. This update also enables all NIC variants, which were
missing from our manually maintained PCI ID whitelist before.
New drivers-nic and drivers-interactive depot packages
======================================================
As already described in section [Unified build directories for ARM],
_drivers_nic_ packages nicely hide the driver configuration internals needed for
a specific board to communicate over the network. Until now there was only one
package available for x86 based PCs. Now, additional _drivers_nic_ packages
are available for:
:boards: imx53_qsb imx6q_sabrelite linux muen pbxa9 rpi zynq
Beside the formerly available _drivers_interactive_ packages for linux, pbxa9
and pc, there are now additional ones for the following:
:boards: imx53_qsb rpi muen
Platforms
#########
For most kernel environments, the core component provides a ROM module named
'platform_info', which comprises information provided by either the kernel or
the bootloader. The information entails e.g., the TSC clock frequency and
framebuffer dimensions. Most of the information is of interest for special
device driver components only.
Over the time, there was an increasing need to incorporate the information
about which kernel Genode runs on top of. Thereby, special test components,
like depot_autopilot could use the information to, e.g., skip certain tests
on kernels known to not support them. Moreover, there are rare corner-cases
where kernels behave differently, for instance, interrupts are enumerated
differently on certain ARM platforms. Rather than maintaining multiple driver
binaries with different names depending on specific kernels, the
'platform_info' ROM module can now be used to differentiate between kernels
when necessary.
Execution on bare hardware (base-hw)
====================================
This release comes with fundamental optimizations and corrections for
executing Genode on bare hardware when using the core component as the actual
kernel.
In the past, we could observe some serious peculiarities regarding the timing
behavior on the hw kernel. After a careful review, we identified the obstacles
that led to time drifts on several platforms and to quite different runtime
execution.
First and foremost, we limited the CPU-load wasted by the kernel, which
unnecessarily made new scheduling decisions quite often. When the hw kernel
was started as an experiment, there was less focus on performance, but more on
simplicity. Instead of caring about state changes that make a scheduling
decision necessary, the scheduler was asked for the next execution context
unconditionally, whenever the kernel was entered. Now, the scheduler gets
invoked only whenever an execution context gets blocked, or unblocked, or if
the kernel's timer fires due to a timeout. This dramatically influences the
CPU-load caused by the hw kernel in a positive way.
The timing accuracy got increased by reworking most hardware timer drivers
used in the kernel to let the timer never stop counting. Moreover, we limit
the scope in between reading the clock and adjusting the next timeout to a
minimum. The whole internal time representation got widened to 64-bit.
In some rare use cases, we could observe components that do I/O polling, and
thereby actively ask for pending signals, to starve. The reason was a gap in
the hw kernel's syscall API. Beside the ability to wait for signals, the
base-library offers the ability to check for pending signals without blocking
in the case of no available signals. The equivalent call in the kernel was
still missing, and is now present and integrated in the base-library of
base-hw.
ARM architecture
----------------
With this release, we add the i.MX 7 Dual SABRE reference board to the rich
hardware zoo Genode runs directly on top of. This includes the use of the
virtualization extensions available on this platform.
Apart from the new board support, several optimizations were added
specifically for the ARM architecture. Several unnecessary cache maintenance
operations were eliminated, which resided in the code base since the time when
the kernel used a separate address-space only. Moreover, the kernel-lock -
used when several execution contexts on different CPU-cores try to enter the
kernel - does not spin anymore. Instead, the CPU goes into a sleep-state to
save energy. As a side-effect, multi-core scenarios become usable when
executed in Qemu.
X86 architecture
----------------
Since the newly used compiler version makes aggressive use of FPU instructions
including the core component, the kernel itself makes use of FPU registers and
state. Therefore, lazy FPU switching becomes a no go for base-hw. Although, we
incorporated eager FPU switching into the ARM-specific part of the hw kernel
already, the x86 version was still missing it. Now, the FPU context of a thread
gets saved and restored on every kernel entry and exit on x86 too.
Updated Muen separation kernel
==============================
The Muen port has been updated to the latest development version, which comes
with many improvements under the hood. Most notably this version of Muen brings
support for Linux SMP subjects, GNAT Community 2018 toolchain support as well
as much improved build speed, which is most noticeable during autopilot runs.
Additionally, the debug server buffer size in the Genode system policy has been
increased to avoid potential message loss in case of rapid successive logging.
_Thanks to Adrian-Ken Rueegsegger of [https://codelabs.ch - Codelabs] for_
_this welcome contribution!_
NOVA microhypervisor
====================
The kernel got updated due to the tool-chain update to GNU G++ 8.3.0.
Additionally, several issues reported by Julian Stecklina regarding FPU and
page-table synchronization got addressed. The kernel memory allocation at boot
time got more flexible to address target machines with fragmented physical
memory. Additionally, the vTLB implementation is no longer used on AMD
machines whenever nested paging is available.
seL4 microkernel
================
With this release, we extend the variety of hardware to run Genode on top of
the seL4 kernel with NXP's i.MX 7 Dual SABRE reference board. To do so, we had
to update the seL4 tools used to craft a bootable ELF image to a state that is
consistent with the currently supported seL4 kernel version 9.0.1.
As a side-effect of this development work, the General Purpose Timer (GPT) used
in the i.MX series can now be used as a timer service component.
Fiasco.OC microkernel
=====================
As with base-hw and seL4, we add the i.MX 7 Dual SABRE reference board to the
list of working hardware for Genode running on top of the Fiasco.OC
microkernel. Moreover, with Fiasco.OC it is now possible to take the first
steps using Genode on the ARM 64-bit architecture. Therefore, we add Raspberry
Pi 3 as a candidate board to be used with Genode/Fiasco.OC. Currently, only
basic tests without peripheral dependencies are supported.
Tooling and build system
########################
Improved handling of missing ports
==================================
The depot tools _tool/depot/create_ and _tool/depot/extract_ now detect and
report all missing third-party sources - called ports - for a given set of
archives at once. Additionally, the user can tell the tools to download and
prepare such missing ports automatically by setting the argument
'PREPARE_PORTS=1'. Please be aware that doing so may cause downloads and
file operations in your _contrib/_ directory without further interaction.
These features make building archives with dependencies to many ports more
enjoyable. If you merely need a list of ports that are missing for your
archives, you can use the new tool _tool/depot/missing_ports_.
For more details you may read the
[https://genodians.org/m-stein/2019-05-21-depot-missing-ports - article on genodians.org].
Automated depot management
==========================
When using the 'import_from_depot' mechanism of the run tool, one frequently
encounters a situation where the depot lacks a particular archive. Whenever
the run tool detects such a situation, it prompts the user to manually curate
the depot content via the _tool/depot/create_ tool. The need for such manual
steps negatively interferes with the development workflow. The right manual
steps are sometimes not straight-forward to find, in particular after
switching between Git branches.
To relieve the developer from this uncreative manual labor, we extended the
run tool with the option '--depot-auto-update' for managing the depot
automatically according to the needs of the executed run script. To enable
this option, use the following line in the build configuration:
! RUN_OPT += --depot-auto-update
If enabled, the run tool automatically invokes the right depot-management
commands to populate the depot with the required archives, and to ensure the
consistency of the depot content with the current version of the source tree.
The feature comes at the price of a delay when executing the run script
because the consistency check involves the extraction of all used source
archives from the source tree. In regular run scripts, this delay is barely
noticeable. Only when working with a run script of a large system, it may be
better to leave the depot auto update disabled.
Please note that the use of the automated depot update may result in version
updates of the corresponding depot recipes in the source tree (recipe hash
files). It is a good practice to review and commit those hash files once the
local changes in the source tree have reached a good shape.

681
doc/release_notes/19-08.txt Normal file
View File

@@ -0,0 +1,681 @@
===============================================
Release notes for the Genode OS Framework 19.08
===============================================
Genode Labs
The stated theme of this year's [https://genode.org/about/road-map - road map]
is "bridging worlds", which expresses our ambition to smoothen the practical
use of Genode-based systems such as Sculpt OS. The current release pays
tribute to this ambition by addressing a great number of practical concerns:
How to accommodate the staggering variety of keyboard layouts out there?
(Section [Flexible keyboard layouts])
How can the system gracefully respond when confronted with exotic USB devices?
(Section [Storage-stack improvements])
How to set the system time from within the system? How does SNTP fit in here?
(Section [General system time concept])
How to approach the remote administration of the system?
(Section [Enhanced SSH terminal])
How to copy and paste text securely between mutually distrusting subsystems?
(Section [Clipboard])
Or how to overcome the captive portal of a Hotel WiFi with Sculpt OS?
(Section [Disposable VM for handling captive portals])
By providing answers to those questions, we believe to make Genode - and Sculpt
OS in particular - generally more useful.
As another take on "bridging worlds", we continue our effort to bring the rich
Sculpt OS software stack to the 64-bit ARM world, in particular to our most
loved SoC family, namely NXP i.MX. Section [64-bit ARM and NXP i.MX8] reports
on our progress in this direction.
Under the hood, there are a few exciting developments that will greatly reduce
the effort of running existing software on Genode. In particular, Genode's
(entirely optional) C runtime has gained the ability to emulate the
traditional execve and fork mechanisms.
(Section [Consolidation of the C runtime and Noux]) This will eventually
alleviate the need for our present noux runtime environment to the benefits of
better performance and increased flexibility.
Further highlights of Genode 19.08 are a major update of Qt5 to version 5.13
(Section [Updated Qt5]) and the continuation of our kernel-agnostic
virtualization story (Section [Virtualization]).
Flexible keyboard layouts
#########################
Genode is used worldwide in a multilingual context beyond Germany and common
technical realms of English. Therefore, we had to address localized
keyboard-input handling for quite some time now and introduced the
_input-filter_ component in
[https://genode.org/documentation/release-notes/17.02#Input-event_filter - 17.02].
The component merges input streams and applies several forms of input
transformations, in particular the application of keyboard layouts to
supplement the input-event stream with character events.
But as we are by no means localization experts, our solution, while performing
a solid job for selected layouts, also had some quirks and rough edges when it
came to French or even Swiss German. First, our oversimplified notion of
[https://en.wikipedia.org/wiki/Caps_Lock - Caps Lock] as _just a pressed Shift_
_key_ is plain wrong but part of all our character-generator configurations.
We just missed this drawback because none of our developers uses Caps Lock
regularly. Further, US English and Germany layouts work very well without
[https://en.wikipedia.org/wiki/Dead_key - dead keys], but crossing any German
border (except the Austrian) is impossible without support for key sequences
composing special characters. The French keyboard layout in Genode tried to
alleviate the lack of compose sequences by adding an additional Circumflex
modifier and character mapping, which unfortunately is not standard.
[image keyboard_stack]
Beginning at this state of affairs, we researched common practice in
international keyboard-input handling, sought a quasi-standard source for
layout configurations, and addressed the drawbacks mentioned before. During
our research we found out that no current implementation is void of critique
and, therefore, decided to look more into X11/XKB as our open-source
quasi-standard solution, but always had an eye on the proprietary world.
The handling of key events in X11/XKB happens on three layers.
:Key codes: On the key-code layer, the device driver programs the
keyboard and generates a stream of key-code (i.e., scan-code)
events, which represent the physical location of the actual key on
the keyboard.
:Key symbols: These key codes are mapped to key symbols, which
represent the label imprinted on the key. So, the key code producing
US English _Q_ (QWERTY keyboard) generates _A_ on a French keyboard
(AZERTY). Modifiers like Shift, AltGr, and Caps Lock are included in
the key-symbol mapping. Additionally, some layouts map key codes to
dead key symbols, which start the before-mentioned compose
sequences. Key repeat is also implemented as key-symbol repeat
actually.
:Characters: On top of this stack, the key symbols are mapped to
characters represented as Unicode codepoints or UTF-8 strings.
The procedure obviously includes key symbols that have no character
representation (e.g. Control and Alt). Key symbols forming a valid compose
sequence generate characters on this level (e.g., dead-key circumflex plus
e generates ê).
We limited our research to Western keyboard-input handling and only had a
blink into the direction of Chinese-Japanese-Korean (CJK) and advanced input
methods (IM). This simplification is supported by the fact that CJK can also
be based on the mechanisms mentioned with some limitations only. Nevertheless,
we do not expect to never touch this topic again.
After doing our homework of keyboard-input handling, we worked on squeezing
all available layout information out of X11/XKB, which resulted in a small
tool residing in _tool/xkb2ifcfg_. For those wondering, the name is just a
silly acronym for _XKB to input-filter_ _configuration_ that pays tribute to
the boringness of this task. After building the tool by a run of 'make' in the
tool path, it can be used as follows. Please make sure you have libxkbcommon
development packages installed beforehand.
! xkb2ifcfg generate <layout> <variant> <locale>
!
! xkb2ifcfg generate us euro en_US.UTF-8
! xkb2ifcfg generate de nodeadkeys de_DE.UTF-8
If the parameter combination is available, xkb2ifcfg prints a input-filer
chargen configuration for the selected layout to standard output. Valid
'layout' and 'variant' options can be figured out from the LAYOUTS section in
'man 7 xkeyboard-config', where 'variant' strings are depicted in parentheses
after the layout (e.g., 'us(euro)'). The 'locale' option has the standard
locale syntax (see /usr/share/i18n/locales). The tool needs all three
parameters to gather the correct key-map and compose-sequence information. The
generated chargen configurations include '<map>' and '<key>' nodes
corresponding to significant modifier states and '<sequence>' nodes (described
later). For simplicity of the generator, the '<key>' nodes always use the
'code="..."' attribute, but also have a comment with the UTF-8 character
appended.
! <key name="KEY_MINUS" code="0x00df"/> <!-- ß -->
Last, we addressed the improvement of the input-filter character generator and
the actual chargen configuration files in Genode. Therefore, we specified the
modifier configuration assumed by the standard chargen files as '<mod1>'
corresponds to Shift, '<mod2>' to Control, '<mod3>' to AltGr, and '<mod4>' to
Caps Lock.
! <mod1> <key name="KEY_LEFTSHIFT"/> <key name="KEY_RIGHTSHIFT"/> </mod1>
! <mod2> <key name="KEY_LEFTCTRL"/> <key name="KEY_RIGHTCTRL"/> </mod2>
! <mod3> <key name="KEY_RIGHTALT"/> </mod3> <!-- AltGr -->
! <mod4> <rom name="capslock"/> </mod4>
As outlined above, the '<key>' nodes generated by xkb2ifcfg always use the
'code' attribute for the Unicode codepoint. Because of this and because UTF-8
also refers to codepoints, we deprecated the 'b0/b1/b2/b3' attributes for
character definition with this release.
The chargen is also extended by the '<sequence>' configuration node. A
sequence node permits the definition of dead-key/composing character
sequences. With such sequences, the character is not generated instantly on
key press but only after the sequence is completed. If an unfinished sequence
can't be completed due to an unmatched character, the sequence is aborted and
no character is generated. We support sequences of up to four characters at
the moment.
For example, the French AZERTY
[https://docs.microsoft.com/en-us/globalization/keyboards/kbdfr.html - keyboard layout]
has a dead key for Circumflex Accent _^_ right of the _P_ key (which is
bracket left _[_ on US keyboards). When Circumflex is pressed no visible
character should be generated instantly but the accent must be combined with a
follow-up character (e.g., Circumflex plus _a_ generates _â_).
Dead keys can be defined in the '<key>' nodes of any '<map>' by using
codepoints not used for direct output, for example, Combining Diacritical
Marks beginning at U+0300. The French Circumflex example can be configured
like follows.
! <mod1>
! <key name="KEY_LEFTSHIFT"/> <key name="KEY_RIGHTSHIFT"/>
! </mod1>
! <map>
! <key name="KEY_Q" code="0x0061"/> <!-- a -->
! <key name="KEY_LEFTBRACE" code="0x0302"/> <!-- dead_circumflex -->
! </map>
! <map mod1="true">
! <key name="KEY_Q" code="0x0041"/> <!-- A -->
! </map>
! <sequence first="0x0302" second="0x0061" code="0x00e2"/> <!-- â -->
! <sequence first="0x0302" second="0x0041" code="0x00c2"/> <!-- Â -->
Fortunately, the configuration is automatically generated by xkb2ifcfg, but
admittedly quite extensive. Therefore, we manually amended the chargen
configurations before adding them to Genode, which also gave us the chance to
apply some adjustments like follows for AltGr in Swiss German.
! <map mod1="false" mod2="false" mod3="true" mod4="false">
! <key name="KEY_1" code="0x00a6"/> <!-- ¦ (*) -->
! <key name="KEY_4" code="0x00b0"/> <!-- ° (*) -->
! <key name="KEY_5" code="0x00a7"/> <!-- § (*) -->
! </map>
Beside the advanced input methods mentioned before, there are still loose ends
we are going to address in the upcoming releases. For example, the current key
handling in our Qt5 back end maps localized key symbols incorrectly (think
AZERTY vs. QWERTY) in combination with shortcuts like Ctrl-A.
64-bit ARM and NXP i.MX8
########################
64-bit ARM support in our custom base-hw kernel
-----------------------------------------------
By introducing rudimentary Raspberry Pi 3 support on top of the Fiasco.OC
kernel in the previous release, the first ARM 64-bit support has entered the
Genode OS framework. We continued pursuing the ARM 64-bit path and introduce
support for Raspberry Pi 3 as well as the i.MX8 evaluation kit (EVK), this
time using our own base-hw kernel.
Noteworthy additions in the base-hw kernel are support for the AARCH64 system
level architecture, and the use of the modern GIC v3 interrupt controller on
top of the i.MX8 EVK board. In comparison to the GICv2, GICv3 adds support for
more than eight CPUs, more than 1020 interrupt IDs, and offers fast register
access to the CPU interface, instead of memory-mapped I/O access. Minor
changes had to be made to the page-table implementation of ARMv7 with Large
Physical Address Extension (LPAE) to re-use it for ARMv8. Moreover, the
internal kernel API for TLB maintenance needed to be changed slightly for all
ARM platforms.
We expanded our regular testing infrastructure with two AARCH64 platforms,
namely Raspberry Pi 3 via Qemu and the NXP i.MX8 EVK board as physical
hardware. Both platforms are driven with a single CPU core only at the moment.
Network driver for i.MX7 and i.MX8
----------------------------------
We updated the 'fec' network driver to version 4.16.3, which adds support for
i.MX7 and i.MX8 SoCs. This makes i.MX8 a viable platform for Genode-based
networking scenarios.
Enhanced packaging and test infrastructure for ARMv8
----------------------------------------------------
Besides the improved base-hw kernel, we enabled additional infrastructure for
ARMv8 platforms. For example, noux packages - like _coreutils_, _bash_ - are
now available, the standard C++ library is in place, and support for Genode's
port of the Linux TCP/IP stack is enabled.
Additionally, ARMv8 is now regularly tested within our nightly
_depot_autopilot_ runs.
Base framework and OS-level infrastructure
##########################################
Tracing
=======
Support for fast tracing has been built into Genode for a long time. However,
the stakes to take advantage of this feature remained high because convenience
functions were not in place. With the current release, we added the support
for easy trace setups through a VFS plugin. The plugin is called _vfs_trace_
and can be mounted into a Genode component as follows:
!<config>
! <vfs>
! <trace ram=32MB/>
! </vfs>
!</config>
This configuration will create a trace file system at the root of the VFS. The
_ram_ attribute is mandatory and determines the maximum size of all trace
buffers. The file system forms a recursive directory structure that represents
the parent/child relationship of running components, whereas the leaf
directories represent single threads within a component. Each leaf directory
currently contains three files:
:'enable': Start or stop the tracing of a thread by writing "true" or "false"
into the file.
:'buffer_size': Allows for the configuration of the trace-buffer size for the
thread in the usual Genode format (e.g. 5M, 512K, 1024).
:'trace_buffer': This read-only file contains the current content of the trace
buffer. Each trace entry can only be read once, after that only new entries
appear. "tail -f" can also be used to display continuous output.
As an example, tracing is started by writing _true_ to the _enable_ file:
! echo "true" > enable
The trace buffer can then be displayed using Unix tools like _tail_
! tail -f trace_buffer
which provides a continuous output.
Additionally, we have added the _trace_ function to _base/log.h_ that
facilitates identical functionality as _Genode::log_
! Genode::trace("Tracepoint value: ", value);
In order to enable tracing, the parent must provide the "TRACE" service. For a
real world example on Sculpt OS, please refer to this
[https://genodians.org/ssumpf/2019-06-18-trace_fs - Genodians article].
With the _vfs_trace_ plugin in place, we removed the outdated _trace_fs_.
Consolidation of the C runtime and Noux
=======================================
On our [https://genode.org/about/road-map#August_-_Release_19.08 - road map],
we vaguely hinted at our plan for the "consolidation" of the noux runtime,
which is actually meant as a polite way of announcing that we are going to
remove it. We introduced the
[https://genode.org/documentation/release-notes/11.02#Noux_-_an_execution_environment_for_the_GNU_userland - Noux runtime]
in 2011 as a way to execute command-line-based GNU software directly on
Genode. It has served us well over the years and is - in fact - a crucial
ingredient of Sculpt OS and other system scenarios such as the Genodians.org
web server. Noux supplements Genode with two valuable assets, namely a
flexible and expandable virtual file system (VFS) layer, and the
implementation of the
[https://genode.org/documentation/release-notes/12.02#Noux_support_for_fork_semantics - Unix way]
to spawn applications ('fork' and 'execve').
In the
[https://genode.org/documentation/release-notes/17.02#Enhanced_VFS_infrastructure - meantime],
noux' VFS implementation has become independent from the noux runtime and is
now prominently employed by Genode's C runtime and the VFS server component.
Genode's C runtime became more and more complete, alleviating the use of noux
as POSIX compatibility layer except for programs that depended on a working
implementation of 'fork' and 'execve'.
The current release fills this remaining gap in Genode's C runtime by
providing 'fork', 'execve', and cousins such as 'wait4' and 'getpid' as
regular parts of the libc. This will eventually make noux redundant.
Note that this change does *NOT* make Genode reliant on POSIX. The C runtime
including the Unix features are entirely optional.
As one stepping stone of this undertaking, noux applications, which previously
had to be compiled for noux, have become binary compatible with the regular C
runtime. So one can execute programs like 'bash' directly as a Genode
component without any friction.
There are a few collateral improvements of Genode's dynamic linker and the C
runtime on the account of the new 'fork' and 'execve' implementation. E.g., in
addition to the already supported 'stdin', 'stdout', and 'stderr'
configuration, the libc can be instructed to initialize arbitrary file
descriptors as follows:
! <config>
! ...
! <libc ...>
! <fd id="3" path="/dev/log" writeable="yes" readable="no" seek="10"/>
! ...
! </libc>
! </config>
The libc-based implementation of 'fork' and 'execve' can be tried out via
the new _ports/run/bash.run_ script. Note that there are still a number of
limitations such as the lack of signal and ioctl handling. Pipes are not
supported, and shebangs ('#!') are not interpreted yet. That said, once those
missing pieces come into place, we can fade out the use of noux within Genode.
General system time concept
===========================
Briefly speaking, up to now there has been no notion of an overall concept of
system time in Genode. Components that need to have access to some kind of
real time are either configured locally, e.g., libc-based components access a
configured "device" (/dev/rtc), which just might be an inline file system
containing an artificial timestamp or the VFS RTC plugin, while other
components query some RTC session directly. Most of the time, this session is
provided by the 'rtc_drv' on x86 machines, which is somewhat costly as reading
the RTC via I/O ports takes time and is therefore done scarcely. For example,
the libc will query an RTC source only once and uses this initial value to
interpolate the current time. However, for executing long-running components,
it will be necessary to adjust the clock to compensate for any occurring clock
drift or to correct a misconfigured clock in general. In addition it is
desirable to be able to use a remote time source, e.g., an NTP-server, to
synchronize the system time.
To address this, we came up with the following concept:
[image system_rtc]
The new "System RTC" component, located at
_repos/libports/src/server/system_rtc_, acts as proxy for the RTC service in
front of the actual RTC driver. It uses the driver to get the initial RTC
value and then uses a timer session (via the timeout framework) to locally
interpolate the time. In contrast to querying the RTC driver, querying the
System RTC is fast.
The RTC driver and the System RTC are bundled up together in the new
_drivers-rtc-pc_ package. The runtime of this package requests two ROM modules
used to update the RTC value. The first one, named 'system_set_rtc', is used
to update the proxy component while the second one, called 'hw_set_rtc', is
used by the RTC driver to write the value into the battery-backed RTC. A
separate component, potentially accessing a remote time source, may generate
these ROMs to adjust the time in the package's runtime.
The new native *SNTP* client at _repos/libports/src/app/sntp_client_ is such a
component. It periodically requests the current time from a given SNTP server
and generates a report. The report produced by the component contains the time
as UTC/GMT. Depending on the system policy, it can be used to update the time
of the System RTC and/or instruct the driver to set the RTC value.
To propagate such changes to RTC values, the RTC session was enhanced by the
new 'set' signal. A client of the session can install a signal handler to
adapt its own time when necessary. Based on this, the time back end of the
libc was changed to instantiate a watch handler for the RTC device, which,
when triggered, will cause the libc to re-read the RTC value.
This constellation should, under normal operation, allow for second to
sub-second granularity updates of the overall system time and avoid drifting
away from network time.
Accessing SMBIOS tables
=======================
The System Management BIOS (SMBIOS) is a specification that allows for reading
management information produced by the BIOS of a system as a collection of
data structures in memory. It has the potential to eliminate the need for the
operating system to probe hardware for discovering present devices and their
characteristics. Nowadays, the SMBIOS specification is implemented widely in
PC systems, which includes modern UEFI systems as well. The data structures
are referred to as _tables_ or _records_ by public documentation.
The new native SMBIOS decoder at _os/src/app/smbios_decoder_ can be used on
x86 to parse SMBIOS tables and report gathered information in a human-readable
way. Besides general table information like number and size of structures,
etc., the component supports complete parsing of SMBIOS structures of types
"BIOS", "System", and "Baseboard".
The component is free from any code for acquiring an SMBIOS table through
means like the bootloader or BIOS information. It expects a table to be
present through a regular Genode ROM session with a 'smbios_table' label. This
way, the underlying system is required to find, select, and save the raw table
on startup and create a ROM module out of it. This is currently achieved on
NOVA and base-hw through an interplay of kernel, the core component, and the
ACPI driver and was tested for legacy BIOSes as well as UEFI systems.
Clipboard
=========
Genode introduced a principle copy-and-paste mechanism already
[https://genode.org/documentation/release-notes/15.11#Copy_and_paste - four years ago].
However, originally created as a part of a tech demo, the mechanism remained
unused in our day to day Genode work. This changed now. We took the
integration of copy-and-paste support in Sculpt OS as an opportunity to revive
and refine the existing mechanism and supplement it with the features needed
to make it practical for daily use. We believe that the result aligns ease of
use nicely with security. The concept is described in a
[https://genodians.org/nfeske/2019-07-03-copy-paste - dedicated article]
at Genodians.org.
On a technical level, the existing clipboard component has received a new
option that allows for dynamic information-flow policies based on user
interactivity (keyboard focus, activity). When setting the config attribute
'match_labels="yes"', the clipboard performs plausibility checks for copy and
paste operations against the focus of the Nitpicker GUI server. All aspects of
the clipboard policy - including information-flow policies - have become
reconfigurable.
To make window-manager clients compatible with the clipboard's dynamic policy,
the window manager got enhanced with the ability to proxy the interaction with
the clipboard. GUI clients in turn - in particular the graphical *terminal* -
became able to interact with the clipboard. With the '<config>' attribute
'copy="yes"' specified, the terminal allows the user to select text to be
reported to a "clipboard" report. The selection mode is activated by holding
the left shift key. While the selection mode is active, the text position
under the mouse pointer is highlighted and the user can select text via the
left mouse button. Upon release of the mouse button, the selection is
reported. Vice versa, with the '<config>' attribute 'paste="yes"' specified,
the terminal allows the user to paste the content of a "clipboard" ROM session
to the terminal client by pressing the middle mouse button.
Finally, we integrated those new abilities into Sculpt OS and into several
installable packages, including virtual machines, the noux-system package,
and graphical Qt5-based applications.
Enhanced SSH terminal
=====================
This release paves the way for remotely managing Genode devices over SSH.
Until now, only interactive SSH sessions were supported. It is now possible to
execute commands from a remote SSH client. E.g., 'ssh noux@localhost -p 5555
"ls -hal /bin/"'. For non-interactive sessions, ssh_terminal requires a helper
component. This component is responsible to create the environment for the
command to run in. You can find an example for such a component at
_gems/src/test/exec_terminal_. It starts noux in a sub init and executes the
provided command inside of it. The new _ssh_exec_channel.run_ script gives a
demonstration on how this feature can be used.
This work is a contribution by Sid Hussmann of
[https://gapfruit.com - Gapfruit]. Thanks for this great new feature!
Storage-stack improvements
==========================
The desire of one Genode developer to exchange data via Iomega ZIP drives
between an Atari Falcon and Sculpt OS called for a number of small
improvements across several components of the storage stack.
First, the USB-block driver has been changed to exit on an initialization
failure instead of waiting for another (supported) device. This change enables
the Sculpt manager to detect such conditions and release the USB device
hardware by removing the driver component. Such a failed initialization may
happen with exotic USB-storage devices such as ZIP drives. With the device
released, however, it can be assigned to a virtual machine to access it using
a guest OS with a broader support of devices.
Second, the USB-block driver received new support for issuing the SCSI
START-STOP command at initialization time, thereby overcoming the ZIP-drive
initialization failure.
Third, we enhanced the part-block component with the ability to parse AHDI
partition schemes and detect the GEMDOS variant of FAT as used by Atari TOS.
Fourth, we enabled the Rump VFS plugin to access GEMDOS file systems. The
GEMDOS variant is readily supported by NetBSD's "msdos" file-system driver.
However, it must explicitly be enabled by a mount flag. Hence, we added the
principle ability for passing mount flags to NetBSD file-system drivers and
enabled the MSDOSFSMNT_GEMDOSFS flag based on the VFS plugin's config
attribute 'gemdos="yes"'.
With these changes in place, data can now be exchanged directly between
Atari-formatted disks and Sculpt OS. That said, advanced use cases such as
media changes at runtime are not covered yet.
Updated Ada/SPARK runtime
=========================
Genode's Ada/SPARK runtime is developed and maintained by
[https://componolit.com - Componolit]. Thanks for this excellent
collaboration!
The updated Componolit Ada runtime 1.1.0 increases the proof coverage and
cleans up the source-code structure. SPARK mode is now enabled wherever
possible and unneeded abstractions have been removed. Furthermore, the 64-bit
addition and subtraction have been proven to be free of runtime errors.
As a new feature, the runtime now supports the use of inline assembly in Ada.
The removal of unneeded features such as the incomplete threading support for
the secondary stack has greatly reduced the runtime's complexity while keeping
the current functionality available. Also GNAT.IO has been removed as its
implementation was incomplete and complex. A simpler replacement has been
introduced with 'Componolit.Runtime.Debug'.
Unrelated to Genode, the runtime now supports [https://muen.sk/ - Muen] and
the API/ABI of the runtime has been separated from the GNAT ABI.
Libraries and applications
##########################
Updated Qt5
===========
We updated our Qt5 port to the latest upstream version 5.13.0. Before
preparing the 'qt5' port, please make sure to build and install the updated
Qt5 host tools with the 'tool/tool_chain_qt5' script.
Virtualization
==============
As follow-up of our work on the
[https://genode.org/documentation/release-notes/19.05#Kernel-agnostic_virtual-machine_monitors - kernel agnostic virtual-machine monitor interface]
on x86, we added principle support to run our port of VirtualBox on
Genode/Fiasco.OC. We write _principle_ support, since we managed to get the
VMM running with Fiasco.OC, but unfortunately not all features required by the
VMM are available using the Fiasco.OC kernel, e.g., guest FPU registers, PDPTE
registers, and task-priority support. In practice this means that the VMs with
Windows and Linux come up to a certain point but will fail later whenever the
guest state runs out of synchronization between VMM and hardware. In contrast,
the Seoul VMM runs fine on Fiasco.OC since it does not depend on the mentioned
missing features.
Our main working items have been the completion of transfer of the available
guest registers and control flow synchronization improvements between VMM and
Fiasco.OC kernel. Additionally, the usage of priorities for VirtualBox's
pthreads in the VMM had to be disabled. Finally, some tests for VirtualBox
with Genode/Fiasco.OC are enabled for nightly regular testing now.
As a side topic, we added support for using the VirtualBox
[https://forums.virtualbox.org/viewtopic.php?f=2&t=82299&start=15 - CPU profile]
feature, which allows for presenting a different CPUID to the VM than the one
of the real CPU. This can help when running Windows 7 on a Kaby Lake or newer
CPU, which are considered _unsupported hardware_ and reason enough not to
receive security updates from Microsoft. The feature can be used on Genode by
adding the 'CpuProfile' attribute to the '<CPU>' XML node in the .vbox file,
like:
! <CPU CpuProfile="Intel Core i7-5600U">
Disposable VM for handling captive portals
==========================================
It is common that WiFi networks require the user to interact with a specific
web page before gaining access to full network functionality. Such captive
portal pages are completely individual to the accessed network and not limited
in the use of common web techniques. Therefore, their handling is best be done
using a fully-featured web browser like Mozilla Firefox.
This is where, in a Genode-based desktop system like Sculpt, a disposable VM
for hosting a minimal browser setup becomes desirable. Its goal is to unlock a
network for the native Genode surroundings with as little inconvenience as
possible just to be thrown away afterwards without any side effects on the
system.
Now, one could use the Firefox appliance VM of Sculpt (see the
[https://genode.org/documentation/release-notes/18.05 - release notes] or the
[https://genodians.org/alex-ab/2019-03-06-disposal-browser-vm - Genodians article])
for this. But this VM aims for a long-term browsing experience which, in the
context of mere captive-portal handling, brings some drawbacks like a much
higher RAM consumption or the required sessions for USB detection and shared
folders.
Furthermore, in the captive portal VM, there's no need for managing windows or
browser tabs. The one browser tab needed can always be shown in fullscreen. It
is also unnecessary for the browser to maintain a content cache or remember
user data. This can reduce resource consumption.
[image captive_portal_vm]
The VM we came up with is provided as package for Sculpt by Martin Stein
(depot user 'mstein'). You'll possibly need to manually add Martin's
[https://github.com/genodelabs/genode/tree/master/depot/mstein - depot key and download location]
to your Sculpt depot directory. After enabling this user, the captive portal
VM can be found in the Sculpt menu under "Depot -> mstein -> Virtual
Machines -> vbox5-nova-captive-portal".
The VM is based on a TinyCore 10 Linux with Xserver, i3 WM, and a tailored
Firefox browser. The package runtime doesn't need access to your file system,
it merely loads some ROMs into a RAM FS, so, it will completely forget any
changes made during a session. Therefore, it's also safe to simply remove an
instance via the Leitzentrale component-view once you don't need it anymore.
The guest additions are also included to make the VM window resizable.
Build system and tools
######################
At Genode Labs, we have used _tool/autopilot_ for the steering of tests in our
Continuous Integration workflow for almost a decade now. This implied various
improvements over the years and with the completion of our work on
[https://genode.org/documentation/release-notes/19.05#Unified_build_directories_for_ARM - unified build directories]
it was time to amend this handy tool once again. Unified build directories
support building all components for one CPU architecture in one directory
saving the build server from the redundant work we previously had with
board-specific directories. With the new notion of boards during builds, the
definition of the target platform when integrating Genode system scenarios is
now a triplet of _CPU architecture_, _board_, and _kernel_. This is reflected
in the new '-t <architecture-board-kernel>' command line option, which
instructs autopilot to generate a build directory for _architecture_ and
execute tests for the _board-kernel_ combination.
! autopilot -t x86_64-pc-sel4 -t x86_64-pc-nova -r log
The known options for '-k kernel' and '-p platform' are still supported with
the small change that the platform must now be defined as
_architecture-board_.
! autopilot -p x86_64-pc -k sel4 -k nova -r log
Autopilot now also documents the hidden feature to propagate custom 'RUN_OPTs'
via the 'RUN_OPT_AUTOPILOT' environment variable to the run tool executed.
Besides that, the tool always appends 'RUN_OPT' with '--autopilot'.
! RUN_OPT_AUTOPILOT="--depot-dir /data/depot" autopilot ...

815
doc/release_notes/19-11.txt Normal file
View File

@@ -0,0 +1,815 @@
===============================================
Release notes for the Genode OS Framework 19.11
===============================================
Genode Labs
On our [https://genode.org/about/road-map - road map] for this year, we stated
"bridging worlds" as our guiding theme of 2019. The current release pays
tribute to this ambition on several accounts.
First, acknowledging the role of POSIX in the real world outside the heavens
of Genode, the release vastly improves our (optional) C runtime with respect
to the emulation of POSIX signals, execve, and ioctl calls. With the line of
work described in Section [C runtime with improved POSIX compatibility], we
hope to greatly reduce the friction when porting and hosting existing
application software directly on Genode.
Second, we identified the process of porting or developing application
software worth improving. Our existing tools were primarily geared to
operating-system development, not application development. Application
developers demand different work flows and tools, including the freedom to use
a build system of their choice.
Section [New tooling for bridging existing build systems with Genode]
introduces our new take on this productivity issue.
Third, in cases where the porting of software to Genode is considered
infeasible, virtualization comes to the rescue. With the current release, a
new virtual machine monitor for the 64-bit ARM architecture enters the
framework. It is presented in Section [Virtualization of 64-bit ARM platforms].
As another goal for 2019, we envisioned a solution for block-level device
encryption, which is a highly anticipated feature among Genode users. We are
proud to present the preliminary result of our year-long development in
Section [Preliminary block-device encrypter].
Preliminary block-device encrypter
##################################
Over the past year, we worked on implementing a block-device encryption
component that makes use of the
[https://en.wikipedia.org/wiki/SPARK_(programming_language) - SPARK]
programming language for its core logic. In contrast to common
block-device encryption techniques where normally is little done besides
the encryption of the on-disk blocks, the _consistent block encrypter (CBE)_
aims for more. It combines multiple techniques to ensure integrity -
the detection of unauthorized modifications of the block-device -
and robustness against data loss. Robustness is achieved by keeping snapshots
of old states of the device that remain unaffected by the further operation of
the device. A copy-on-write mechanism (only the differential changes to the
last snapshot are stored) is employed to maintain this snapshot history with
low overhead. To be able to access all states of the device in the same manner,
some kind of translation from virtual blocks to blocks on the device is needed.
Hash-trees, where each node contains the hash of its sub-nodes, combine the
aspect of translating blocks and ensuring their integrity in an elegant way.
During the tree traversal, the computed hash of each node can be easily checked
against the hash stored in the parent node.
The CBE does not perform any cryptography by itself but delegates
cryptographic operations to another entity. It neither knows nor cares about
the used algorithm. Of all the nodes in the virtual block device (VBD), only
the leaf nodes, which contain the data, are encrypted. All other nodes, which
only contain meta-data, are stored unencrypted.
Design
------
As depicted in Figure [cbe_trees], all information describing the various
parts of the CBE is stored in the superblock. The referenced VBD is a set of
several hash trees, each representing a certain device state including the
current working state. Only the tree of the current working state is used to
write data to the block device. All other trees represent snapshots of older
states and are immutable. Each stored device state has a generation number
that provides the chronological order of the states.
As you can see, in the depicted situation, there exist four device states - the
snapshot with generation 3 is the oldest, followed by two newer snapshots and
generation 6 that marks the working state of the virtual device. The tree with
generation 6 is the current working tree. Each tree contains all changes done
to the VBD since the previous generation (for generation 6 the red nodes). All
parts of a tree that didn't change since the previous generation are references
into older trees (for generation 6 the gray nodes). Note that in the picture,
nodes that are not relevant for generation 6 are omitted to keep it manageable.
The actual data blocks of the virtual device are the leaf nodes of the trees,
shown as squares.
[image cbe_trees]
Whenever a block request from the client would override data blocks in
generation 6 that are still referenced from an older generation, new blocks for
storing the changes are needed. Here is where the so-called _Free Tree_ enters
the picture. This tree contains and manages the spare blocks. Spare blocks are
a certain amount of blocks that the CBE has in addition to the number of blocks
needed for initializing the virtual device. So, after having initialized a
virtual device, they remain unused and are only referenced by the Free Tree.
Therefore, in case the VBD needs new blocks, it consults the Free Tree (red
arrow).
In the depicted situation, writing the first data block (red square) would
require allocating 4 new blocks as all nodes in the branch leading to the
corresponding leaf node - including the leaf node itself - have to be written.
In contrast, writing the second data block would require allocating only one
new block as the inner nodes (red circles) now already exist. Subsequent write
requests affecting only the new blocks will not trigger further block
allocations because they still belong to the current generation and will be
changed in-place. To make them immutable we have to create a new snapshot.
The blocks in generation 5 that were replaced by the change to generation 6
(blue nodes) are not needed for the working state of the virtual device
anymore. They are therefore, in exchange for the allocated blocks, added to
the Free Tree. But don't be fooled by the name, they are not free for
allocation yet, but marked as "reserved" only. This means, they are
potentially still part of a snapshot (as is the case in our example) but the
Free Tree shall keep checking, because once all snapshots that referenced the
blue blocks have disappeared, they become free blocks and can be allocated
again.
To create a new snapshot, we first have to make all changes done to the VBDs
working state as well as the Free Tree persistent by writing all corresponding
blocks to the block-device. After that, the new superblock state is written to
the block-device. To safeguard this operation, the CBE always maintains several
older states of the superblock on the block device. In case writing the new
state of the superblock fails, the CBE could fall back to the last state that,
in our example, would contain only generations 3, 4, and 5. Finally, the
current generation of the superblock in RAM is incremented by one (in the
example to generation 7). Thereby, generation 6 becomes immutable.
A question that remains is when to create snapshots. Triggering a snapshot
according to some heuristics inside the CBE might result in unnecessary
overhead. For instance, the inner nodes of the tree change frequently during a
sequential operation. We might not want them to be re-allocated all the time.
Therefore, the creation of a snapshot must be triggered explicitly from the
outside world. This way, we can accommodate different strategies, for
instance, client-triggered, time-based, or based on the amount of data
written.
When creating a snapshot, it can be specified whether it shall be disposable
or persistent. A disposable snapshot will be removed automatically by the CBE
in two situations, either
* When there are not enough usable nodes in the Free Tree left to
satisfy a write request, or
* When creating a new snapshot and all slots in the superblock that might
reference snapshots are already occupied.
A persistent snapshot, or quarantine snapshot, on the other hand will never be
removed automatically. Its removal must be requested explicitly.
During initialization, the CBE selects the most recent superblock and reads the
last generation value from it. The current generation (or working state
generation) is then set to the value incremented by one. Since all old blocks,
that are still referenced by a snapshot, are never changed again, overall
consistency is guaranteed for every generation whose superblock was stored
safely on disk.
Implementation
--------------
Although we aimed for a SPARK implementation of the CBE, we saw several
obstacles with developing it in SPARK right from the beginning. These obstacles
mainly came from the fact that none of us was experienced in designing
complex software in SPARK. So we started by conducting a rapid design-space
exploration using our mother tongue (C++) while using only language features
that can be mapped 1:1 to SPARK concepts. Additionally, we applied a clear
design methodology that allowed us to keep implementation-to-test cycles
small and perform a seamless and gradual translation into SPARK:
* _Control flow_
The core logic of the CBE is a big state machine that doesn't block. On each
external event, the state machine gets poked to update itself accordingly.
C++ can call SPARK but SPARK never calls C++. The SPARK code therefore
evolves as self-contained library.
* _Modularity_
The complex state machine of the CBE as a whole is split-up into smaller
manageable sub-state-machines, working independently from each other. These
modules don't call each other directly. Instead, an additional superior
module handles the interplay. This is done by constantly iterating over all
modules with the following procedure until no further progress can be made:
# Try to enter requests of other modules into the current one
# Poke the state machine of the current module
# The current module may have generated requests - Try to enter them into
the targeted modules
# The current module may have finished requests - Acknowledge them at the
modules they came from
Each module is represented through a class (C++) respectively a package with
a private state record (SPARK).
* _No global state_
There are no static (C++) or package (SPARK) variables. All state is kept in
members of objects (C++) respectively records (SPARK). All packages are pure
and sub-programs have no side-effects. Therefore, memory management and
communication with other components is left to OS glue-code outside the
core logic.
This approach worked out well. Module by module, we were able to translate the
C++ prototype to SPARK without long untested phases, rendering all regression
bugs manageable. In Genode, the CBE library is currently integrated through
the CBE-VFS plugin. Figure [cbe_modules] depicts its current structure and the
integration via VFS plugin.
[image cbe_modules]
The green and blue boxes each represent an Ada/SPARK package. The translation
to SPARK started at the bottom of the picture moving up to the more abstract
levels until it reached the Library module. This is the module that handles
the interplay of all other modules. Its interface is the front end of the CBE
library. So, all green packages are now completely written in SPARK and
together form the CBE library. Positioned above, the CXX library in blue is
brought in by a separate library and exports the CBE interface to C++. This
way, the CBE can also be used in other environments including pure SPARK
programs. The CXX Library package is not written in SPARK but Ada and performs
all the conversions and checks required to meet the preconditions set by the
SPARK packages below.
At the C++ side, we have the VFS plugin. Even at this level, the already
mentioned procedure applies: The plugin continuously tries to enter requests
coming from the VFS client (above) into the CBE (below), pokes the CBE state
machine, and puts thereby generated block/crypto requests of the CBE into the
corresponding back-ends (left). This process is repeated until there is no
further progress without waiting for an external event.
Current state
-------------
In its current state, the CBE library is still pretty much in flux and is not
meant for productive use.
As the Free Tree does not employ copy-on-write semantics for its meta-data, a
crash, software- or hardware-wise, will corrupt the tree structure and renders
the CBE unusable on the next start.
This issue is subject to ongoing work. That being said, there are
components that, besides being used for testing, show how the interface of the
CBE library lends itself to be integrated in components in different ways. At
the moment, there are two components making use of the CBE library as
block-device provider.
The first one is the aforementioned CBE-VFS plugin. Besides r/w access to the
working tree and r/o access to all persistent snapshots, it also provides a
management interface where persistent snapshots can be created or discarded.
Its current layout is illustrated by Figure [cbe_vfs]. The VFS plugin
generates three top directories in its root directory. The first one is the
_control_ directory. It contains several pseudo files for managing the CBE:
[image cbe_vfs]
:'key': set a key by writing a string into the file.
:'create_snapshot': writing 'true' to this file will attempt to create
a new snapshot. (Eventually the snapshot will
appear in the 'snapshots' directory if it could be
created successfully.)
:'discard_snapshot': writing a snapshot ID into this file will discard
the snapshot
The second is the 'current' directory. It gives access to the current
working tree of the CBE and contains the following file:
:'data': this file represents the virtual block device and gives
read and write access to the data stored by the CBE.
The third and last is the 'snapshots' directory. For each persistent snapshot,
there is a sub-directory named after the ID of the snapshot. This directory,
like the 'current' directory, contains a 'data' file. This file, however,
gives only read access to the data belonging to the snapshot.
The CBE-VFS plugin itself uses the VFS to access the underlying block device.
It utilizes the file specified in its configuration. Here is a '<vfs>'
snippet that shows a configured CBE-VFS plugin where the block device is
provided by the block VFS plugin.
! <vfs>
! <dir name="dev">
! <block name="block"/>
! <cbe name="cbe" block="/dev/block"/>
! </dir>
! </vfs>
An exemplary ready-to-use run script can be found in the CBE repository
at _run/cbe_vfs_snaps.run_. This run script uses a bash script to
automatically perform a few operations on the CBE using the VFS plugin.
Afterwards it will drop the user into a shell where further operations
can be performed manually, e.g.:
! dd if=/dev/zero of=/dev/cbe/current/data bs=4K
The second component is the CBE server. In contrast to the CBE-VFS plugin,
it is just a simple block-session proxy component that uses a block connection
as back end to access a block-device. It provides a front-end block session to
its client, creates disposable snapshots every few seconds, and uses the
'External_Crypto' library to encrypt the data blocks using AES-CBC-ESSIV. The
used key is a plain passphrase. The following snippet illustrates its
configuration:
! <start name="cbe">
! <resource name="RAM" quantum="4M"/>
! <provides><service name="Block"/></provides>
! <config sync_interval="5" passphrase="All your base are belong to us"/>
! </start>
The _run/cbe.run_ run script in the CBE repository showcases the use of the
CBE server.
Both run scripts will create the initial CBE state in a RAM-backed
block device that is then accessed by the CBE server or the CBE-VFS
plugin.
The run-script and the code itself can be found on the
[https://github.com/cnuke/cbe/tree/cbe_19.11 - cbe/cbe_19.11] branch on
GitHub. If you intend to try it out, you have to checkout
the corresponding
[https://github.com/cnuke/genode/tree/cbe_19.11 - genode/cbe_19.11]
branch in the Genode repository as well.
Future plans
------------
Besides addressing the current shortcomings and getting the CBE library
production-ready so that it can be used in Sculpt, there are still
a few features that are currently unimplemented. For one we would like
to add support for making it possible to resize the VBD as well as the
Free Tree. For now the geometry is fixed at initialization time and cannot
be changed afterwards. Furthermore, we would like to enable re-keying,
i.e., changing the used cryptographic key and re-encrypting the tree
set of the VBD afterwards. In addition to implementing those features, the
overall tooling for the CBE needs to be improved. E.g., there is currently
no proper initialization component. For now, we rely on a component
that was built merely as a test vehicle to generate the initial trees.
Virtualization of 64-bit ARM platforms
######################################
Genode has a long history regarding support of all kinds of
virtualization-related techniques including
[https://genode.org/documentation/release-notes/9.11#Paravirtualized_Linux_on_Genode_OKL4 - para-virtualization],
[https://genode.org/documentation/articles/trustzone - TrustZone],
hardware-assisted virtualization on
[https://genode.org/documentation/articles/arm_virtualization - ARM],
[https://genode.org/documentation/release-notes/13.02#Full_virtualization_on_NOVA_x86 - x86],
up to the full virtualization stack of
[https://genode.org/documentation/release-notes/14.02#VirtualBox_on_top_of_the_NOVA_microhypervisor - VirtualBox].
We regard those techniques as welcome stop-gap solutions for using non-trivial
existing software stacks on top of Genode's clean-slate OS architecture. The
[https://genode.org/documentation/release-notes/19.05#Kernel-agnostic_virtual-machine_monitors - recent]
introduction of a kernel-agnostic interface to control virtual machines (VM)
ushered a new level for the construction respectively porting of
virtual-machine monitors (VMM). By introducing a new ARMv8-compliant VMM
developed from scratch, we continue this line of work.
The new VMM builds upon our existing proof-of-concept (PoC) implementation for
ARMv7 as introduced in release
[https://genode.org/documentation/release-notes/15.02#Virtualization_on_ARM - 15.02].
In contrast to the former PoC implementation, however, it aims to be complete
to a greater extent. Currently, it comprises device models for the following
virtual hardware:
* RAM
* System Bus
* CPU
* Generic Interrupt Controller v2 and v3
* Generic Timer
* PL011 UART (limited)
* Pass-through devices
The VMM is able to load diverse 64-bit Linux kernels including
Device-Tree-Binary (DTB) and Initramfs. Currently, the implementation uses a
fixed memory layout for the guest-physical memory view, which needs to be
reflected by the DTB used by the guest OS. An example device-tree source file
can be found at _repos/os/src/server/vmm/spec/arm_v8/virt.dts_. The actual VMM
is located in the same directory.
Although support for multi-core VMs is already considered internally, it is
not yet finished. Further outstanding features that are already in development
are Virtio device model support for networking and console. As the first - and
by now only - back end, we tied the VMM to the ARMv8 broadened Kernel-agnostic
VM-session interface as implemented by Genode's custom base-hw kernel. As a
side effect of this work, we consolidated the generic VM session interface
slightly. The RPC call to create a new virtual-CPU now returns an identifier
for identification.
The VMM has a strict dependency on ARM's hardware virtualization support
(EL2), which comprises extensions for the ARMv8-A CPU, ARM's generic timer,
and ARM's GIC. This rules out the Raspberry Pi 3 board as a base platform
because it does not include a GIC but a custom interrupt-controller without
hardware-assisted virtualization of interrupts. To give the new VMM a try, we
recommend using the run script _repos/os/run/vmm_arm.run_ as a starting point
for executing the VMM on top of the i.MX8 Evaluation Kit board.
New tooling for bridging existing build systems with Genode
###########################################################
Genode's development tools are powerful and intimidating at the same time.
Being designed from the perspective of a whole-systems developer, they put
emphasis on the modularity of the code base (separating concerns like
different kernels or system abstraction levels), transitive dependency
tracking between libraries, scripting of a wide variety of system-integration
tasks, and the continuous integration of complete Genode-based
operating-system scenarios. Those tools are a two-edged sword though.
On the one hand, the tools are key for the productivity of seasoned Genode
developers once the potential of the tools is fully understood and leveraged.
For example, during the development of Sculpt OS, we are able to
change an arbitrary line of code in any system component and can test-drive
the resulting Sculpt system on real hardware within a couple of seconds.
As another example, the almost seamless switching from one OS kernel to
another has become a daily routine that we just take for granted without
even thinking about it.
On the other hand, the sophistication of the tools stands in the way of
application developers who are focused on a particular component instead
of the holistic Genode system. In this case, the powerful system-integration
features remain unused but the complexity of the tools and the build system
prevails. Speaking of build systems, this topic is ripe of emotions
anyway. _Developers use to hate build systems._ Forcing Genode's build
system down the throats of application developers is probably not the best
idea to make Genode popular.
This line of thoughts prompted us to re-approach the tooling for Genode from
the perspective of an application developer. The intermediate result is a new
tool called Goa:
:Goa project at GitHub:
[https://github.com/nfeske/goa]
Unlike Genode's regular tools, Goa's work flow is project-centered. A project
is a directory that may contain source code, data, instructions how to
download source codes from a 3rd party, descriptions of system scenarios, or
combinations thereof. Goa is independent from Genode's regular build system.
It combines Genode's package management (depot) with commodity build systems
such a CMake. In addition to building and test-driving application software
directly on a Linux-based development system, Goa is able to aid the process
of exporting and packaging the software in the format expected by Genode
systems like Sculpt OS.
At the current stage, Goa should be considered as work in progress. It's a new
approach and its success is anything but proven. That said, if you are
interested in developing or porting application software for Genode, your
feedback would be especially valuable. As a starting point, you may find the
following introductory article helpful:
:Goa - streamlining the development of Genode applications:
[https://genodians.org/nfeske/2019-11-25-goa]
Base framework and OS-level infrastructure
##########################################
File-system session
===================
The file-system session interface received a much anticipated update.
Writing modification times
--------------------------
The new operation WRITE_TIMESTAMP allows a client to update the modification
time of a file-system node. The time is defined by the client to keep
file-system servers free from time-related concerns. The VFS server implements
the operation by forwarding it to the VFS plugin interface. At present, this
new interface is implemented by the rump VFS plugin to store modification
times on EXT2 file systems.
Enhanced file-status info
-------------------------
The status of a file-system node as returned by the 'File_system::Status'
operation has been revisited. First, we replaced the fairly opaque "mode" bits -
which were an ad-hoc attempt to stay compatible with Unix - with the explicit
notion of 'readable', 'writeable', and 'executable' attributes. We completely
dropped the notion of users and groups. Second, we added the distinction
between *continuous* and *transactional* files to allow for the robust
implementation of continuous write operations across component boundaries. A
continuous file can be written-to via a sequence of arbitrarily sized chunks
of data. For such files, a client can split a large write operation into any
number of smaller operations in accordance to the size of the used I/O
buffers. In contrast, a write to a transactional file is regarded as a
distinct operation. The canonical example of a transactional file is a
socket-control pseudo file.
Virtual file-system infrastructure
==================================
First fragments of a front-end API
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The VFS is mostly used indirectly via the C runtime. However, it is also
useful for a few components that use the Genode API directly without any
libc. To accommodate such users of the VFS, we introduced the front-end
API at _os/vfs.h_ that covers a variety of current use cases. Currently, those
use cases revolve around the watching, reading, and parsing of files and
file-system structures - as performed by Sculpt's deployment mechanism.
Writing to files is not covered.
Improved file-watching support
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
All pseudo files that use the VFS-internal 'Readonly_value_file_system'
utility have become able to deliver watch notifications. This change enables
VFS clients to respond to VFS-plugin events (think of terminal resize)
dynamically.
Speaking of the *terminal VFS plugin*, the current release enhances the plugin
in several respects. First, it now delivers status information such as the
terminal size via pseudo files. Second, we equipped the VFS terminal file
system with the ability to detect user interrupts in the incoming data stream,
and propagate this information via the new pseudo file '.terminal/interrupts'.
Each time, the user presses control-c in the terminal, the value stored in
this pseudo file is increased. Thereby, a VFS client can watch this file to
get notified about the occurrences of user interrupts.
VFS plugin for emulating POSIX pipes
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
We added a new VFS plugin for emulating POSIX pipes. The new plugin creates
pipes between pairs of VFS handles. It replaces the deprecated libc_pipe
plugin. In contrast to the libc_pipe plugin, which was limited to pipes within
one component, the new VFS plugin can also be used to establish pipes between
different components by mounting the plugin at a shared VFS server.
C runtime with improved POSIX compatibility
===========================================
Within Genode, we used to think of POSIX as a legacy that is best avoided.
In fact, the foundational components of the framework do not depend on a
C runtime at all. However, higher up the software stack - at the latest when
3rd-party libraries enter the picture - a working C runtime is unavoidable. In
this statement, the term "working" is rather muddy though. Since we have never
fully embraced POSIX, we were content with cutting corners here and there. For
example, given Genode's architecture, supporting 'fork' and 'execve' seemed
totally out of question because those mechanisms would go against the grain of
Genode.
However, our growing aspiration to bridge the gap between existing popular
applications and Genode made us re-evaluate our stance towards POSIX.
All technical criticism aside, POSIX is immensely useful because it is
a universally accepted stable interface. To dissolve friction between
Genode and popular application software, we have to satisfy the application's
expectations. This ignited a series of developments, in particular
the added support for 'fork' and 'execve' - of all things - in
[https://genode.org/documentation/release-notes/19.08#Consolidation_of_the_C_runtime_and_Noux - Genode 19.08],
which was nothing short of surprising, even to us.
The current release continues this line of development and brings the
following improvements.
Execve
------
The libc's 'execve' implementation got enhanced to evaluate the path of the
executable binary according to the information found on the VFS, in particular
by traversing directories and following symbolic links. This enables the libc
to execute files stored at sub directories of the file system.
Furthermore, 'execve' received handling for *executing shell scripts* by
parsing the shebang marker at the beginning of the executable file. This way,
the 'execve' mechanism of the libc reaches parity with the feature set of the
Noux runtime that we traditionally used to host Unix software on top of
Genode.
Modification-time handling
--------------------------
By default, the libc uses the just added facility for updating the timestamp
of file-system nodes when closing a written-to file, which clears the path
towards using tools like 'make' that rely on file-modifications times.
The libc's mechanism can be explicitly disabled by specifying
! <libc update_mtime="no"...>
This is useful for applications that have no legitimate access to a time
source.
Emulation of 'ioctl' operations via pseudo files
------------------------------------------------
With the current release, we introduce a new scheme of handling ioctl
operations, which maps 'ioctl' calls to pseudo-file accesses, similar to how
the libc already maps socket calls to socket-fs operations.
A device file can be accompanied with a (hidden) directory that is named after
the device file and hosts pseudo files for triggering the various device
operations. For example, for accessing a terminal, the directory structure
looks like this:
! /dev/terminal
! /dev/.terminal/info
! /dev/.terminal/rows
! /dev/.terminal/columns
! /dev/.terminal/interrupts
The 'info' file contains device information in XML format. The type of the XML
node corresponds to the device type. Whenever the libc receives a TIOCGWINSZ
ioctl for _/dev/terminal_, it reads the content of _/dev/.terminal/info_ to
obtain the terminal-size information. In this case, the _info_ file looks as
follows:
! <terminal rows="25" columns="80/>
Following this scheme, VFS plugins can support ioctl operations by providing
an ioctl directory in addition to the actual device file.
Emulation of POSIX signals
--------------------------
Even though there is no notion of POSIX signals at the Genode level, we
can reasonably emulate certain POSIX signals at the libc level. The current
release introduces the first bunch of such emulated signals:
:SIGWINCH: If 'stdout' is connected to a terminal, the libc watches the
terminal's ioctl pseudo file _.terminal/info_. Whenever the terminal
size changes, the POSIX signal SIGWINCH is delivered to the application.
With this improvement, Vim becomes able to dynamically adjust itself
to changed window dimensions when started as a native Genode component
(w/o the Noux runtime environment).
:SIGINT: If 'stdin' is connected to a terminal, the libc watches the
terminal's pseudo file _.terminal/interrupts_. Since, the terminal VFS
plugin modifies the file for each occurred user interrupt (control-c),
the libc is able to reflect such an event as SIGINT signal to the
application.
:Process-local signal delivery: The libc's implementation of 'kill' got
enhanced with the ability to submit signals to the local process.
Support for arbitrarily large write operations
----------------------------------------------
The number of bytes written by a single 'write' call used to be constrained by
the file's underlying I/O buffer size. Even though our libc correctly returned
this information to the application, we found that real-world applications
rarely check the return value of 'write' because partial writes do usually not
occur on popular POSIX systems. Thanks to the added distinction between
continuous and transactional files as described in Section
[File-system session], we became able to improve the libc's write operation to
iterate on partial writes to continuous files until the original write count
is reached. The split of large write operations into small partial writes as
dictated by the VFS infrastructure becomes invisible to the libc-using
application.
Input-event handling
====================
In Genode 19.08, we undertook a comprehensive rework of our keyboard-event
handling in the light of localization and also promised to tie up remaining
loose ends soon.
First, we again dived into our character generators for a thorough check of
our stack of keyboards and fixed remaining inconsistencies in French and
German layouts. En passant, we also increased the default RAM quotas for
the input filter to 1280K in our recipes to cope with the increased
layout-configuration sizes in corner cases.
Next - and more importantly - we subdued the monsters lurking in our Qt5
keyboard back end and enabled transparent support for system-wide keyboard
layout configuration for Qt5 components. One important change during this work
was to move the handling of control key sequences into the clients. For
example, the graphical terminal and Qt5 interpret key events in combination
with the CTRL modifier based on characters and, thus, support CTRL-A with
AZERTY and QWERTY layouts correctly. As a result we removed all CTRL modifier
(mod2) configurations from our character-generator configurations.
Finally we'd like to point out one important change of our rework that
repeatedly led to surprises: For keys without character mappings the reworked
character-generator mechanism emits invalid codepoints in contrast to
codepoints with value 0. For that reason, components interpreting character
events should check 'Codepoint::valid()' to prevent the processing of invalid
characters (and not the frequent pattern of 'codepoint.value != 0').
NIC router
==========
The NIC router has received the ability to report the link state of its NIC
interfaces (downlinks and uplinks). To control this mechanism, there are two
new boolean attributes 'link_state' and 'link_state_triggers' in the <report>
tag of the NIC router configuration. If the former is set to "true", the report
will contain the current link state for each interface:
! <domain name="domain1">
! <interface label="uplink1" link_state="false"/>
! <interface label="downlink1" link_state="true"/>
! </domain>
! <domain name="domain2">
! <interface label="downlink2" link_state="true"/>
! </domain>
The second attribute decides whether to trigger a report update each time the
link state of an interface changes. By default, both attributes are set to
"false".
Device drivers
==============
Platform driver on x86
~~~~~~~~~~~~~~~~~~~~~~
During our enablement of Genode on a
[https://genodians.org/chelmuth/2019-10-21-sculpt-elitebook - recent notebook],
we spotted some PC platform shortcomings, we address with this release. Most
prominently we added support for
[https://en.wikipedia.org/wiki/PCI_configuration_space#Bus_enumeration - 64-bit PCI base address registers]
to the x86 platform driver. This allows the use of PCI devices that are
assigned to physical I/O-memory regions beyond 4 GiB by the boot firmware.
Wireless driver
~~~~~~~~~~~~~~~
We added the firmware images for the 5000 and 9000 series of Intel wireless
devices to the firmware white-list in the _wifi_drv_ component. Such devices
as 5100AGN, 5300AGN and 5350AGN as well as 9461, 9462 and 9560 should now be
usable on Genode.
Libraries and applications
##########################
VirtualBox improvements
=======================
The GUI handling of our VirtualBox port got improved to react on window-size
changes more instantly. The effect is that an interactive adjustment of the
window size, e.g., on Sculpt, becomes quickly visible to the user. Still, the
VM may take some time to adjust to the resolution change, which ultimately
depends on the behavior of the driver of the VirtualBox guest additions.
Updated 3rd-party software
==========================
With the addition of the 64-bit ARM architecture (AARCH64) with the
[https://genode.org/documentation/release-notes/19.05#Broadened_CPU_architecture_support_and_updated_tool_chain - 19.05]
release, it became necessary to update the libraries the Genode tool chain
(gcc) depends on in order to support AARCH64 properly. This concerns the GNU
multi precision arithmetic library (gmp), which has been updated from version
4.3.2 to 6.1.2, as well as the libraries that depend on it: Multi precision
floating point (mpfr) and multi precision complex arithmetic (mpc). All those
old versions did not offer support for the AARCH64 architecture, which is a
requirement to make Genode self hosting. Targets for building binutils and GCC
within Genode for AARCH64 are in place, GNU make is in place, and even code
coverage (gcov) has been added. This work puts AARCH64 in line with other
supported CPU architectures and emphasizes our interest in the ARM 64-bit
architecture.
Platforms
#########
Execution on bare hardware (base-hw)
====================================
With the previous release, Genode's base-hw kernel got extended to support the
ARMv8-A architecture in principle. The first hardware supported was the
Raspberry Pi 3 as well as the i.MX8 evaluation kit (EVK). But only a single
CPU-core was usable at that time. Now, we lifted this limitation. On both
boards, all four CPU-cores are available henceforth.
Removed components
##################
The current release removes the following components:
:gems/src/app/launcher:
The graphical launcher remained unused for a few years now. It is not
suitable for systems as flexible as Sculpt OS.
:os/src/app/cli_monitor:
CLI monitor was a runtime environment with a custom command-line interface
to start and stop subsystems. It was part of the user interface of our
first take on a Genode-based desktop OS called
[https://genode.org/documentation/release-notes/15.11#Genode_as_desktop_OS - Turmvilla].
Nowadays, we use standard command-line tools like Vim to edit init
configurations dynamically, which is more flexible and - at the same time -
alleviates the need for a custom CLI. The CLI-monitor component was too
limited for use cases like Sculpt anyway.
Along with the CLI monitor, we removed the ancient (and untested for long
time) _terminal_mux.run_ script, which was the only remaining user of the CLI
monitor.
:fatfs_fs, rump_fs, and libc_fatfs plugin:
The stand-alone file-system servers fatfs_fs and rump_fs as well as the
fatfs libc plugin have been superseded by the fatfs and rump VFS plugins.
The stand-alone servers can be replaced by using the VFS server plus the
corresponding VFS plugin as a drop-in replacement.

814
doc/release_notes/20-02.txt Normal file
View File

@@ -0,0 +1,814 @@
===============================================
Release notes for the Genode OS Framework 20.02
===============================================
Genode Labs
This year's [https://genode.org/about/road-map - road map] is all about making
Genode and Sculpt OS more approachable. It turns out that the first release of
the year already pays tribute to that goal. First, it equips Sculpt OS with a
much more logical and welcoming graphical user interface
(Section [Redesign of the administrative user interface of Sculpt OS]).
Second, it greatly reduces the friction when hosting existing applications on
Genode by smoothening several rough edges with respect to POSIX compatibility,
and by generally improving performance.
Most topics of the release are closely related to Sculpt. The biggest
break-though is certainly the ability of running Sculpt OS on 64-bit ARM
hardware (Section [Sculpt OS on 64-bit ARM i.MX8 hardware]) along with our
custom virtual machine monitor (VMM). On PC hardware, Sculpt users can enjoy
an updated audio driver and optimizations of the Seoul VMM. Furthermore,
Sculpt's window manager received the much anticipated ability to use virtual
desktops.
At the framework-API level, the most significant changes are the introduction
of dedicated types for inter-thread synchronization patterns
(Section [Base-framework refinements]) and a new library for
bringing the benefits of the Genode architecture to the application level
(Section [New sandbox library based on the init component]).
Redesign of the administrative user interface of Sculpt OS
##########################################################
On our [https://genode.org/about/road-map - road map] for 2020, we stated
the reducing of the barrier of entry as our main concern of the year.
We highlighted the ease of use of Sculpt OS as one particular work area.
Removing Unix from the picture
------------------------------
Until now, Sculpt's administrative user interface - lyrically called
Leitzentrale - employed a small Unix runtime and the Vim editor as utility for
basic file operations and for the tweaking of configurations. Even though this
was a practical intermediate solution, we have to face the fact that not
everyone loves the Unix command-line interface as much as we do. Quite the
opposite, actually. When presenting Sculpt, we can clearly sense that people
with a non-Unix background are put off by it. The audience generally loves the
runtime graph, visual cues, and discoverability. Furthermore, command-line
interfaces are (albeit wrongly) perceived as archaic and impenetrable relics
by many computer users who are otherwise perfectly happy with the notion of
files and directories. We identified that file-manipulation tasks performed in
the Leitzentrale are rare and simple. Relying on Unix for those basic tasks is
like taking a sledgehammer to crack a nut. On average, the Leitzentrale is
used in just a few moments a day for basic things like browsing a file-system
hierarchy, glimpsing at the reports stored on the report file system, deleting
or copying a file or two, or tweaking a configuration file. With a Unix shell
presenting one barrier, Vim is certainly an even higher one. Familiarity with
Vim should definitely not be a prerequisite for using an operating system.
Following this reasoning, we decided to swap out the command-line interface
and Vim by a simple GUI-based file browser and a notepad-like editor, which do
not require any learning curve.
Note that even once the Unix command-line interface is removed from Sculpt's
Leitzentrale, advanced users will still be able to manipulate Sculpt's config
file system via a Unix runtime deployed as a regular component, similar to the
use of the noux-system package we have today.
New user-interface layout
-------------------------
The move away from the command-line interface goes hand in hand with the
redesign of the overall user-interface layout. A new panel at the top of the
screen contains two centered tabs for switching between the runtime graph and
the file-system browser.
[image sculpt_20.02_panel]
The storage-management functionality has been moved from the former storage
dialog into the respective nodes of the runtime graph. E.g., to format a block
device, the user can now select a USB or storage node of the graph to get a
menu of block-device-level operations.
[image sculpt_20.02_storage]
The network-management is now located at a drop-down menu that can be toggled
via a button at the right side of the panel.
[image sculpt_20.02_network]
A new button on the left side of the panel allows the user to toggle a
drop-down menu for GUI settings. At the current time, there is only the option
to adjust the font size. In the future, the dialog will give easy access to
the screen-resolution options and the keyboard layout.
The log-message view is now hidden in another drop-down menu that can be
toggled via a panel button. So when starting the system, the user is greeted
with only the runtime graph, which is a much nicer and cleaner looking
experience.
Informative or diagnostic messages are displayed in the left-bottom corner of
the screen.
[image sculpt_20.02_message]
The "Files" tab of the panel switches the main screen area to a simple file
browser that lists all file systems available. By toggling one of the
file-system buttons, the directory hierarchy can be browsed. When hovering
a file, an "Edit" or "View" button appears, which can be used to open
the file in a text area that appears on the right side of the file browser.
The editor supports the usual notepad-like motions, operations, and
shortcuts (control-c for copy, control-v for paste, control-s for save).
[image sculpt_20.02_editor]
Half-way there
--------------
With the current release, one can already accomplish a lot without having to
resort to a command-line interface: connecting to the network, managing
storage devices, installing and deploying software, inspecting the system
state, and tweaking configurations.
There are still a few gaps though. In particular the file browser does
not yet support file operations like the copying, renaming, or removal of
files. For these tasks, the current version of Sculpt still features the
Unix-based inspect window, which can be accessed by toggling the "Inspect"
button inside the USB or storage dialog. Once selected, the panel presents an
"Inspect" tab that features the familiar Unix shell and Vim. Note, however,
that we keep the inspect window only as an interim solution. It will
eventually be removed. As with every new feature, there are still rough edges
to be expected in the editor and file browser, e.g., the editing of files with
long lines or the browsing of directories with many entries is not
appropriately covered yet.
To see the current new version of Sculpt OS in action, you may find the
following presentation entertaining.
:Live demonstration of Sculpt OS at FOSDEM 2020:
[https://fosdem.org/2020/schedule/event/uk_sculpt/]
The new version 20.02 of Sculpt OS is part of this release and can be built
from source and used right now. Several Genode developers already provide
ready-to-use packages for the new version. The software depots by alex-ab,
cnuke, skalk are worth exploring. A downloadable system image along with an
updated manual will be released shortly.
Sculpt OS on 64-bit ARM i.MX8 hardware
######################################
Within the past two releases, big steps were taken to support ARMv8 hardware in
the Genode OS framework. After implementing basic support for Raspberry Pi 3,
and the i.MX 8M Evaluation Kit, the network card was enabled for the latter.
Moreover, we updated the Linux TCP/IP, and C library ports, as well as
the Noux environment to support the architecture. Finally, with the latest
releases, a new ARMv8-compliant virtual-machine monitor for the base-hw kernel
entered the framework.
The rapid achievements motivated us to strive for a more ambitious scenario to
run on top of the currently focused ARMv8 hardware platform. So why not using
Sculpt OS on the i.MX 8M System-on-Chip?
Persistent storage
==================
There were several challenges to cope with initially. First, persistent
storage was needed. Luckily, the Genode OS framework contained already an
SD-card driver implementation for the i.MX series. The driver was written for
Genode from scratch and initially supported the i.MX53 SoC only. From then, it
got extended repeatedly to drive the SD-card controller of several i.MX6 and
i.MX7 platforms. Therefore, it was not a big issue to support the new hardware
too. However, when we later used it in Sculpt, it turned out that the driver
has some low-latency requirements. If those were not met, it got stuck. This
was the time where the CPU-quota mechanism came in handy in a real-world
scenario. It helped to let the interrupt handler of the driver be scheduled in
time, and thereby let the driver run stable.
Having a working block device is one part, but it is of little use without a
file system. In Sculpt OS, the NetBSD rump kernel's ext2 file-system is
typically used to host the depot package system and for keeping configuration
files persistent. Unfortunately, the version of NetBSD as used in Genode's
rump kernel port does not contain the ARMv8 architecture. Of course, we could
have upgraded the rump kernel as a whole. But this software stack is quite
complex with a lot of threads reproducing a sophisticated state machine. It
took some time in the past to meet its required semantics. Therefore,
backporting some header definitions and a few architecture-dependent functions
seemed more attractive. Luckily, it turned out to be the right decision, and
after a day of backporting work, the file system could run on ARMv8.
Display engine
==============
One of the more challenging tasks was certainly the enabling of the Display
Controller Subsystem (DCSS) of the i.MX 8M SoC. Originally, we hoped to profit
from our experiences with the Image Processing Unit (IPU), the display engine
of former i.MX SoCs. But as it turned out, the DCSS is a completely new
design, and has not much in common with the IPU. When first writing a driver
for the IPU of the i.MX53, we were surprised by the complexity and flexibility
of this piece of hardware. Back then, it took months to get something
meaningful working. To not lose too much time by re-implementing a driver from
scratch, we decided to take the DDE Linux approach, which worked out pretty
fast. The resulting driver should provide the same flexibility like the Linux
original one. However, as the i.MX 8M EVK board provides a HDMI connector
only, we did not test more than that. The configuration of the driver is
analogous to the Intel framebuffer driver, and looks like the following:
! <config>
! <connector name="HDMI-A-1" width="1920" height="1080" hz="60" enabled="true"/>
! </config>
Later, when using the driver in practice within the Sculpt OS, we could
experience a slightly sluggish behaviour, which was due to a missing
architectural back end of the blitting library of Genode. After tweaking this
too, the graphical user interface experience was good.
USB and Input
=============
The last missing I/O device to run Sculpt OS on the ARMv8 was something for
user generated input. Therefore, the existent USB host controller driver for
the i.MX series got updated. The only roadblock here was the powering of the
device. As there is no platform driver for the target hardware yet, which
would manage power and clocks, the hardware either has to be pre-configured
correctly, or the driver has to enable it on its own. Ethernet card, SD-card,
and the display engine were all already powered by the bootloader, but not
USB. In contrast to the first devices, the u-boot bootloader turns off USB
explicitly as soon as it starts the OS. As an interim solution, we patched
u-boot to not turn off the USB host controller, and enforced u-boot to
initialize the powering in our boot scripts. Therefore, if one wants to use
USB on the i.MX 8M EVK, make sure to take our modified version. As a
convenient solution, you can use the 'uboot' port within the base repository.
Just issue the following command in the Genode directory:
! tool/ports/prepare_port uboot
Finally, you have to copy u-boot to the SD-card as root user:
! dd if=`tool/ports/current uboot`/imx8q_evk/imx-mkimage/iMX8M/flash.bin \
! of=/dev/sd<?> bs=1k seek=33 conv=fsync
Of course, you have to replace 'sd<?>' with the correct device node of your
attached SD-card.
After enabling the USB host controller driver, we could successfully re-use the
USB HID client driver to drive keyboard and mouse connected to the board. As a
nice side-effect, the list of possible storage devices got extended with USB
mass storage too by adding the USB block client driver.
Missing libraries
=================
Finally, when building the necessary and optional packages for Sculpt OS, we
stumbled across several libraries that needed to be adapted to compile and
link for ARMv8 too. Mostly, the inclusion of some other compilation units and
headers was sufficient. The related libraries are: libssl, libcrypto, libpng,
and Mesa. With the latter two, it is now even possible to execute Qt5
components on the target hardware.
Apart from all the new driver components and extended libraries, the Sculpt
manager had to be slightly modified to execute on the i.MX 8M hardware. In its
original form it is inherently dependent on x86 drivers, as it for example
generates configurations for some of those drivers. For the time being, the
changes to the Sculpt manager are not yet part of the official release.
Nevertheless, you can produce a Sculpt OS image to be run on an i.MX 8M EVK
board by using the following
[https://github.com/skalk/genode/commits/sculpt_20.02_imx8q_evk - topic branch].
Alternatively, you can also have a look at Sculpt OS on ARMv8 hardware by
following the video recordings of the following talk at FOSDEM 2020.
:Live demonstration of Sculpt OS on i.MX 8M EVK at FOSDEM 2020:
[https://fosdem.org/2020/schedule/event/uk_genode_armv8/]
Base framework and OS-level infrastructure
##########################################
New sandbox library based on the init component
===============================================
The init component is Genode's canonical mechanism for the composition of
components. This role was further amplified when init became
[https://genode.org/documentation/release-notes/17.02#Dynamically_reconfigurable_init_component - dynamically reconfigurable].
The latter change cleared the ground for system scenarios like Sculpt OS, the
on-target deployment of packages, and dynamic device discovery. One typical
pattern found in such scenarios is one dynamically configured instance of init
accompanied by a controlling component that is usually called "manager". The
manager would consume reports of the subsystem hosted within the dynamic init,
and adjust the init configuration according to a domain-specific policy. Such
a configuration change, in turn, may trigger new reports, which effectively
turns this setting into a feedback control loop.
Whereas this established pattern is suitable for many scenarios, it is not
always natural. In particular if the manager does not only need to
manage a subsystem but also wants to intercept a service used by the
subsystem, the roles are no longer clear-cut. A practical example is a
GUI application that employs the menu-view component for the GUI rendering
while processing keyboard events locally. This application would need to
intercept the menu-view's GUI session to obtain the stream of user input
events. For such an application, the most natural approach would be the
co-location of the init functionality with the application logic into a
single all-encompassing component.
To accommodate such scenarios where a domain-specific management component is
tightly coupled with a dynamic subsystem, we extracted the child-management
functionality from the init component into a new library called "sandbox". The
library API is located at
[https://github.com/genodelabs/genode/blob/master/repos/os/include/os/sandbox.h - os/include/os/sandbox.h].
In addition to the hosting of components, the sandbox API also allows for the
interaction with the sandboxed children by providing locally implemented
services. The latter mechanism is illustrated by a new test available at
_os/src/test/sandbox_.
POSIX compatibility improvements
================================
During the release cycle of Genode 20.02, we continued our mission to host
POSIX software effortlessly as Genode components. In particular, we followed
up the line of work pursued with the two previous releases
[https://genode.org/documentation/release-notes/19.08#Consolidation_of_the_C_runtime_and_Noux - 19.08] and
[https://genode.org/documentation/release-notes/19.11#C_runtime_with_improved_POSIX_compatibility - 19.11]
with respect to the traditional Unix mechanisms fork, execve, and pipes.
After covering several edge cases - cloexec, file-descriptor lifetimes,
line-buffer handling, vfork, just to name a few - as needed by programs like
make, bash, and tclsh, we eventually reached a state where the website
generator of [https://genodians.org] works without the need for the now
deprecated Noux runtime.
For years we have been running complex software stacks like the Qt-based web
browser on top of our C runtime but not without carefully placed tweaks and
occasional patches. With the current release, we address the area of
concurrency and introduce a thorough reimplementation of the synchronization
primitives namely POSIX mutexes and condition variables as well as semaphores.
We also reaped the fruit of our labor by replacing our custom Qt thread back
end by the standard POSIX-thread based implementation. Further, we reduced the
number of threads in Qt applications by moving the QPA event handling to the
component entrypoint and removing the timed-semaphore utility from LibC.
Beyond Qt, we also address synchronization issues revealed by running a
third-party port of [https://grpc.io/ - gRPC] in our network back ends and
amended thread-local errno in the C runtime. Finally, our POSIX thread
implementation supports cleanup handlers now.
Base-framework refinements
==========================
Replacing the 'Lock' type by new 'Mutex' and 'Blockade' types
-------------------------------------------------------------
Up to now, Genode's lock implementation supports mainly two flavours of usage.
On the one hand, it is used to protect critical sections where the lock is
initialized as unlocked. In the contention case, the lock holder is supposed
to release the critical section. On the other hand, the lock is used as
blockade to synchronize startup between various executions of threads. Here
the lock is initialized as locked during instantiation whereby the thread that
releases the lock is not necessarily the same thread as the creator of the
lock.
We decided to make the two usage patterns more obvious by introducing two
separate classes, called 'Mutex' and 'Blockade'. The reasons are twofold.
First, during code review, the usage pattern at hand becomes more obvious.
Second, by codifying the programmer's intent behind the use of a
synchronization primitive, Genode becomes able to perform additional checks,
and diagnose certain dead-lock situations and other usage errors on the spot.
The separation got introduced shortly before this release. Up to now, it is
only used in 'Genode::Thread', 'Genode::Heap', and 'Genode::Registry'. The
plan is to cultivate the usage across all Genode sources over the next
releases and to ultimately remove the 'Genode::Lock' from the public API.
The 'Mutex' class is more restrictive compared to the 'Lock' class.
* At initialization time, it is always unlocked.
* To enter and leave a critical section the methods 'acquire()' and
'release()' are used.
* A 'Mutex::Guard' is provided, which will 'acquire()' a mutex at
construction time and release it automatically at destruction time of
the guard.
* No thread is permitted to lock twice. The code will generate a warning if
a dead-lock is detected.
* Only the lock holder is permitted to release the mutex. The code will
generate a warning and will not release the mutex if this rule is violated.
! Genode::Mutex mutex;
! mutex.acquire();
! mutex.release();
!
! {
! Genode::Mutex::Guard guard(mutex) /* acquire() during construction */
! } /* release() on guard object destruction */
!
! Genode::Mutex::Guard guard(mutex);
! mutex.acquire(); /* <-- Will cause a warning about the dead-lock */
The 'Blockade' class is always initialized as locked and provides the methods
'block()' and 'wakeup()'. Beside the initialization aspect, the 'Blockade'
behaves up to now like the 'Genode::Lock' implementation.
! Genode::Blockade blockade;
!
! /* step */ /* thread A */ /* thread B */
! 0: -start thread B-
! 1: ... -startup-
! 2: blockade.block(); ...
! 3: -sleep- ...
! 4: -sleep- blockade.wakeup();
! 5: ... ...
Performance optimization of the XML parser
------------------------------------------
Genode's XML parser used to rely on C++ exceptions while parsing, which is an
almost historic artifact inherited from the initial implementation. The
performance penalties of exceptions in the rare use of XML was acceptable
back when we started. But modern Genode systems like Sculpt OS rely on the
dynamic processing of XML like a back bone. The overhead became particularly
apparent when executing [Sculpt OS on 64-bit ARM i.MX8 hardware]. Prompted by
this observation, we reworked the code such that exceptions are no longer
thrown in any hot code path. The public interface of 'Xml_node' remains
unchanged.
New polling variant for register framework
------------------------------------------
Genode's register framework has offered a 'wait_for' method for a long time.
This function sleeps for a certain amount of microseconds and checks if one or
more given conditions become true. The number of attempts to sleep and check
the conditions must also be specified. In case the conditions are not met
after these attempts, a polling timeout exception is thrown. The function
simply returns in case of success. With the current Genode release, we have
added a 'wait_for_any' method with almost the same semantics but instead of
waiting for all conditions to become true, it returns if any condition is
met, and thus, implements a logical OR.
Migration to modern block-device API
====================================
With release 19.02, Genode introduced two new APIs for block-session handling.
The client side of a block session now uses the job API in order to send block
requests to the server, which in turn receives those jobs as requests through
the Request API. These two APIs replace Genode's 'Block::Driver' and
'Block::Session_component' implementations that used the packet stream API
directly, which turned out to be error prone for block session implementations.
Instead, these new APIs wrap the packet stream handling in a controlled
manner while handling all corner cases and even the overcommit of packets.
With the current release, we have adapted Genode's AHCI driver and partition
manager to these new interfaces, with the plan to adjust all block session
clients/servers to the new APIs with Genode release 20.05.
During this line of work, the AHCI driver received a major cleanup. For
example, dynamic memory allocations were removed, the whole initialization
state machine has been removed, ATAPI support for Qemu has been re-enabled,
and Exynos5 AHCI support is gone - since the platform is outdated and not
supported by Genode any more.
Updated audio driver based on OpenBSD 6.6
=========================================
In this release, we updated the 3rd-party sources of the audio driver component
to OpenBSD 6.6 and adapted the emulation glue code. While doing so, we fixed
a bug regarding the 'delay()' implementation where the function expects
microseconds but was given milliseconds. This led to a increased start-up
time of the component. We also fixed the logging back end that accidentally
was rendered silent and brought in the 'printf' back end from DDE Linux to
be able to produce better formatted LOG messages in the future.
Until now the component only supported HDA and EAP (ES1370 PCI) devices. The
first is primarily intended to be used with real hardware whereas the latter
was used during the initial porting effort in Qemu. That being said, the EAP
driver apparently also works on hardware according to community feedback.
Since the HDA driver does not work when used in VirtualBox and users expressed
the desire to also use audio when running in a VM, we enabled another driver,
for which a device-model in VirtualBox exists: the AC97 ICH. As it turned out,
using this driver, we can produce audio, albeit the quality is far from
usable. Nevertheless, with the driver enabled, interested parties are free to
investigate the cause for the current issues.
All in all, this update is solely a catch up effort to stay more
up-to-date with the upstream changes and to pull in HDA quirks for more
recent systems. More interesting changes to the driver component, like
reworking the OpenBSD kernel emulation layer and bringing support for USB
audio devices, are scheduled for future releases.
Support for unlabeled LOG output
================================
In situations where a Genode system is remotely controlled and monitored,
it is useful to allow a special component to produce log output with no
Genode label applied. This way, such a component can produce log data in
a format that is immediately suitable for a controller. This feature can be
enabled for a component by rewriting the label of the component's LOG session
to "unlabeled".
! <route>
! <service name="LOG"> <parent label="unlabeled"/> </service>
! ...
! </route>
Libraries and applications
##########################
Custom virtual machine monitor on ARM
=====================================
The ARMv8-compliant virtual-machine monitor introduced in the previous release
19.11 now contains new device models to enable the interaction with a
virtual-machine via network and terminal services. The new virtual ethernet
card and console implementations are compliant to the virtualization standard
VIRTIO 1.1.
Currently, the VMM cannot be configured to contain specific devices. It is
hard-wired to provide exactly:
* One virtual ethernet card that connects to Genode's "Nic" service,
* A VIRTIO console that opens up a session to the "Terminal" service using the
label "console", and
* The traditional PL011 serial device model, which connects to a
"Terminal" service too but uses the label "earlycon"
Seoul VMM
=========
During the usage of Seoul on Sculpt, it became apparent that the Seoul VMM
caused a constant CPU load even when the guest VM was idling. After some
investigation it became clear that having a fixed rate to synchronize the
guest graphic memory with the Genode GUI service was the main reason for the
constant load. With this release, we added the feature to dynamically adjust
the GUI refresh rate depending on the rate of user interactivity.
Additionally, if all virtual CPUs go to idle state, the GUI refresh is stopped
completely. With these measures, the overall CPU load could be reduced
noticeably.
TCP terminal
============
The TCP terminal is a long-living component in the Genode OS framework since
release 11.11. It can be used, e.g., to connect to a headless Genode system
via telnet. Until now, it always listened to incoming network connections at
configured ports. The port had to be configured for each terminal session
client.
The TCP terminal got extended to either listen to incoming network
connections, or to directly connect to another network server, dependent on
the policy defined for the corresponding terminal client. The following
example configuration illustrates the differences:
! <config>
! <policy label="client" ip="10.0.0.5" port="1234"/>
! <policy label="another_client" port="4567"/>
! </config>
If only a port is described in the policy, the TCP terminal will listen on
that port for incoming connections. If an IP address is provided additionally,
it connects to the IP address using the given port.
Virtual desktops
================
Genode's GUI stack enables a high degree of flexibility. Beside the fundamental
nitpicker component, responsible for basically multiplexing input events and
framebuffer content, there is the window-manager component, and example
implementations of a window-layouter, and decorator. The interplay of the
latter three allows a window management that scales from simple to rich and
sophisticated without lowering its security properties. For a brief description
of its architecture, please refer to the release notes of
[https://genode.org/documentation/release-notes/14.08 - 14.08].
In this architecture, the window layouter is responsible for the arrangement
of the different windows. It exports a data model of the window layout.
Although, the example implementation of the window layouter introduced in
14.08 was simple, it already contained a notion of having different virtual
screens and screen sections, beside the actual window placements. However,
until now there was no use-case of switching dynamically between different
virtual screens respectively window sets related to them.
While using more and more different graphical components within Sculpt, the
window layouter in its initial form hit a limit. Although it already allowed to
switch in-between different windows via configured key-combinations, it became
inconvenient when having more than a handful windows hiding each other.
Therefore, the window layouter now got extended to allow switching dynamically
in between several pre-defined virtual screens. For the time being, one has to
assign a new window to a screen in the rule-set of the window layouter
initially by hand. Defining the currently visible screen can either be done by
editing the rule-set, or by using pre-configured key-combinations.
The new default configuration of the window layouter as exported by its
corresponding depot package looks like the following:
! <config rules="rom">
! <rules>
! <screen name="screen_1"/>
! <screen name="screen_2"/>
! <screen name="screen_3"/>
! <screen name="screen_4"/>
! <screen name="screen_5"/>
! <screen name="screen_6"/>
! <screen name="screen_7"/>
! <screen name="screen_8"/>
! <screen name="screen_9"/>
! <screen name="screen_0"/>
! <assign label_prefix="" target="screen_1" xpos="any" ypos="any"/>
! </rules>
!
! <press key="KEY_SCREEN">
! <press key="KEY_ENTER" action="toggle_fullscreen"/>
! <press key="KEY_1" action="screen" target="screen_1"/>
! <press key="KEY_2" action="screen" target="screen_2"/>
! <press key="KEY_3" action="screen" target="screen_3"/>
! <press key="KEY_4" action="screen" target="screen_4"/>
! <press key="KEY_5" action="screen" target="screen_5"/>
! <press key="KEY_6" action="screen" target="screen_6"/>
! <press key="KEY_7" action="screen" target="screen_7"/>
! <press key="KEY_8" action="screen" target="screen_8"/>
! <press key="KEY_9" action="screen" target="screen_9"/>
! <press key="KEY_0" action="screen" target="screen_0"/>
! ...
As can be seen, individual keys are assigned to switch to a specific virtual
screen. By default ten screens are defined that are accessible via the number
keys. The first screen definition in the rules configuration marks the
currently visible screen.
Menu-view widget renderer
=========================
The line of work described in Section
[Redesign of the administrative user interface of Sculpt OS] called for
the enhancement of Genode's GUI-rendering component. This component - named
menu view - was
[https://genode.org/documentation/release-notes/14.11#New_menu_view_application - originally introduced in Genode 14.11]
for the rendering of the relatively simple menus of an application launcher.
Its software design largely deviates from the beaten track of established
widget toolkits, which come in the form of client-side libraries. The
menu view is not a complete toolkit but solely a dialog renderer sandboxed
in a dedicated component. This design reinforces the strict separation of the
view from the application logic, fosters screen-resolution independence, and -
most importantly - keeps the complexity of pixel processing out of the
application program. Because of the latter, it lends itself to the
implementation of security-sensitive interactive applications.
It would certainly be misguiding to tout our menu-view as feature competitive
with existing toolkits. We certainly won't recommend using it over Qt in
general. But Sculpt's custom administrative user interface "Leitzentrale"
presented us with the perfect playground to explore and grow the potential of
our novel approach.
In contrast to the previous iteration of the Leitzentrale GUI, which relied on
a small Unix runtime and Vim for editing text files, the new version ought to
feature a simple text editor integrated in the GUI. A text editor requires
a much tighter interplay between the view and the actual program logic
compared to an application with just a bunch of buttons. Think about cursor
handling, scrolling text, displaying textual selections, or placing a text
cursor with the mouse. On the course of the work towards the text-area
component featured in the new Leitzentrale, the menu view received the
following improvements:
:Text-cursor support:
The label widget gained the ability to display one or multiple text cursors,
as illustrated by the following example:
! <label text="...">
! <cursor at="10"/>
! </label>
For the display of multiple cursors, each cursor must feature a distinctive
'name' attribute.
:Character position featured in the hover report:
The hovering information provided by the menu view used to be at the
granularity of widgets, which is insufficient for placing a text cursor with
the mouse. Hence, the information of a hovered label additionally provides
the character position within the label now.
:Unquoting label text attribute values:
The text displayed in label widgets is provided by a 'text' attribute value,
which raises the question of how to present '"' characters on the GUI. With
the new version, the attribute value can contain XML-quoted characters,
specifically "&quot;".
:Support for displaying text selections:
Similarly to the way of how a <cursor> can be defined for a <label>
widget, a selection can now be expressed as follows:
! <label ...>
! <selection at="2" length="12"/>
! </label>
:Support of multiple '<float>' widgets within a '<frame>':
We refined the hover reporting of <float> widgets such that a float widget
never responds to hovering unless a child is hovered. This way, it becomes
possible to stack multiple float widgets within one frame and still reach
all child widgets. This is useful for aligning multiple widgets within one
screen area independently from each other. For example, for left-aligning,
centering, and right-aligning the elements of a panel.
:Enforcing the minimum size of a label:
The new '<label min_ex="..">' attribute can be used to enforce a minimum
width in the unit of the size of the character 'x'. In the absence of a
'text' attribute, the minimum height of a label is implicitly set to 0. The
combination of both changes makes the label usable as a horizontal spacer.
:Basic support for styling labels:
The new version allows for the customization of the text color and alpha
value of the label widget by the means of a style-definition file. The
mechanism is exemplified with the new "invisible" label style that sets the
alpha value to zero.
With these few incremental changes in place, the menu-view widget renderer
becomes usable as the basis of the simple text editor used in Sculpt's new user
interface.
Self-hosting the tool chain on 64-bit ARM
=========================================
With our ongoing ARM 64-bit effort, we have successfully updated Genode's tool
chain with release
[https://genode.org/documentation/release-notes/19.05#Broadened_CPU_architecture_support_and_updated_tool_chain - 19.05].
With the current release, we have additionally managed to make Genode's tool
chain self hosting on ARM 64-bit, which means the tool chain can compile
source code on ARM 64-bit directly.
Platforms
#########
Execution on bare hardware (base-hw)
====================================
The generic code base of the base-hw kernel underwent several cosmetic changes
to reduce or eliminate the application of certain problematic constructs like
too much inheritance, pointers, and dynamic casts. Those changes were
motivated to ease the translation of several kernel parts to the Ada/SPARK
language in the context of the Spunky project. For more information regarding
this experiment to write a Genode kernel in Ada/SPARK, please have a look at
the recent [https://genodians.org/m-stein/index - genodians.org article series]
of Martin Stein or listen to his recent
[https://video.fosdem.org/2020/AW1.125/ada_spunky.mp4 - FOSDEM talk].
Moreover, the IPC path implementation got simplified to lower the overhead
costs introduced by the transfer of capabilities. Together with the mentioned
Spunky cleanup efforts, this change measurably improved IPC performance.
The base-hw kernel now exports time consumption of individual threads via
the trace service analogously to the implementation for NOVA. Thereby, it
becomes possible to use for instance the top component within the Sculpt OS
also on this kernel.
Until now, support for the Raspberry Pi 3 was limited to Qemu emulation only.
Thanks to a contribution of Tomasz Gajewski, it is now possible to execute
Genode on all four CPUs of the actual hardware concurrently.
Execution on Linux
==================
Traditionally, the Linux version of Genode serves us as very handy development
vehicle but it was never intended as an actual target platform. On Linux,
Genode is usually executed as a multi-process application on top of a regular
GNU/Linux desktop distribution by specifying 'KERNEL=linux' and 'BOARD=linux'
to the run tool.
However, thanks to the work of Johannes Kliemann, Genode has become able to
run on a bare-bone Linux kernel without any other user land.
We blatantly used to refer to this idea as the
[https://genode.org/about/challenges#Platforms - microkernelization of Linux].
Johannes picked up the idea, supplemented Genode's core with the services
needed for user-level device drivers (IRQ, IOMEM, IOPORT) and supplemented
the tooling for the integration of Genode scenarios into a bootable initrd
image. This target of execution can be addressed by specifying 'KERNEL=linux'
and 'BOARD=pc' to the run tool now. If specified, the run tool will produce a
bootable Linux system image for the given run script and run it in Qemu.
That said, as this line of work is still considered as an experimental
playground - not for productive use - the work flow is not entirely automated.
In particular, one needs to prepare a suitable
[https://github.com/jklmnn/linux/commits/genode - Linux kernel] manually.
If you are interested in the topic, please refer to the background information
given in the [https://github.com/genodelabs/genode/pull/2829 - issue tracker].

Some files were not shown because too many files have changed in this diff Show More