blog/content/2017/12/18/varlink.md

+++
title = "Varlink"
date = 2017-12-18T12:22:38+00:00
tags = ["varlink", "fedora", "programming"]
+++

For a long time we tried to solve the early boot IPC problem. IPC problem you ask? Well, in the early boot phase we
cannot talk D-BUS, because the daemon is not running yet, but to bring up the system, so that the D-BUS daemon can run
(mount stuff, load modules, start other services), we need some kind of IPC. Therefore all the early boot daemons have a
fallback IPC via unix domain sockets with its own homegrown protocol.

<!-- more -->

### K-DBUS ###

Moving the D-BUS daemon in the kernel was our first approach, but with all the functionality the D-BUS daemon
has (signal broadcasting, etc.), there was no clean solution possible, which satisfied all kernel devs.

### BUS1 ###

BUS1 was written, but turned out overly complex and was just a transport without a protocol.

### Varlink

While BUS1 was developed as a transport, our team was looking for a protocol and tried simplifying the D-BUS protocol.
After a lot of iterations not even the variant data protocol was left. Instead Kay Sievers and Lars Karlitski came up
with a super simple JSON based IPC protocol, which only needs a point-to-point connection, so BUS1 was not even needed.

## KISS

The [varlink](http://varlink.org/) protocol is so simple, that language bindings can be implemented easily and using
varlink does not result into spending more time coding for IPC than for the actual problem.
Bindings for
[C](https://github.com/varlink/libvarlink),
[Javascript](https://github.com/varlink/org.varlink.http),
[Go](https://github.com/varlink/org.varlink.http),
[Java](https://dentrassi.de/2017/12/17/varlink-for-java-what-wonderful-world-it-could-be/),
[Python](https://github.com/varlink/python-varlink) 
and [Rust](https://github.com/varlink/rust-varlink) were quickly implemented (some still only as a proof of concept). 
[A service for resolving interface](https://github.com/varlink/org.varlink.resolver) names into connection 
URLs and starting the executable has been written, which resembles some of the D-BUS functionality of resolving 
and activating services.

## APIs

The most important feature of varlink though, is in the interface definition files. They are human readable and can 
be even discussed amongst people, which are not developing the implementation. They enable a "checks and balance" 
system for product management, customers, quality engineering and software developers. Interface stability and 
backwards compatibility can be enforced easily.
To quote the varlink [Ideals page](https://github.com/varlink/documentation/wiki/Ideals):

> ### Simplicity ###
>
> Varlink aims to be as simple as possible. It is not specifically optimized for anything else but ease-of-use and maintainability.
>
> ### [](https://github.com/varlink/documentation/wiki/Ideals#discoverability)Discoverability ###
>
> Varlink services describe themselves; with a machine-readable interface definition, and human-readable documentation.
>
> ### [](https://github.com/varlink/documentation/wiki/Ideals#remotability)Remotability ###
>
> It should be easy to forward, proxy, redirect varlink interfaces over any connection-oriented transport. Varlink should be free of any side-effects of local APIs. All interactions need to be simple messages on a network, not carrying things like file descriptors or references to locally stored files.
>
> ### [](https://github.com/varlink/documentation/wiki/Ideals#focus)Focus ###
>
> Varlink is the protocol and definition of interfaces, but it does not define or provide any significant functionality itself.
>
> ### [](https://github.com/varlink/documentation/wiki/Ideals#errors)Errors ###
>
> Varlink errors should carry enough information to be consumed by machines and automated rules.
>
> ### [](https://github.com/varlink/documentation/wiki/Ideals#testing)Testing ###
>
> Making it easy to run automated tests against varlink interfaces should be considered a necessity when designing interfaces.

Right now the API to a Linux system consists of:
* kernel: ioctls, syscalls, procfs, sysfs
* CLI: options, stdout output to be parsed
* D-BUS, Protobuf, various other IPC, homegrown IPC protocols, …

All these can be replaced by varlink interfaces (yes, even the [kernel interfaces](https://github.com/varlink/linux-varlink)).
Of course varlink is the [15th xkcd standard](https://xkcd.com/927/) here :-P If the adoption of varlink takes off,
then the collection of interfaces could form a common Linux System API.

These interface definitions can be inspected even at runtime via a common interface named `org.varlink.service`,
which every service provides. With this mechanism it is very easy for interpreter languanges
to create bindings at runtime, as we will see in the next chapter.

## Varlink for Python

Ready for some example on how to use varlink with python? First we need some packages to install on Fedora: 
```console
$ sudo dnf copr enable "@varlink/varlink"
$ sudo dnf install fedora-varlink
$ sudo setenforce 0 # needed until systemd is able to create sockets in /run
$ sudo systemctl enable --now org.varlink.resolver.socket
$ varlink help
```

In this example, we will call an interface called `com.redhat.system.accounts` which has the following varlink interface: 
```varlink
interface com.redhat.accounts

type Account (
    name: string,
    uid: int,
    gid: int,
    full_name: string,
    home: string,
    shell: string
)

# Retrieve a list of account information for all known accounts
method GetAll() -> (accounts: []Account)
# Retrieve the account information for a specific user ID
method GetByUid(uid: int) -> (account: Account)
# Retrieve the account information
method GetByName(name: string) -> (account: Account)
# Add new account
method Add(account: Account) -> (account: Account)

error NotFound ()
error CreationFailed (field: string)
```
 
From reading the interface definition, the API should be understandable,
even for people, who have never seen a varlink interface file.

Now, let's call the service from some python code:
```python
from varlink import Client
client = Client(resolve_interface='com.redhat.accounts')
accounts = client.open('com.redhat.accounts', namespaced=True)
ret = accounts.GetByName("root")
print(ret)
```

will output:

```python
namespace(account=namespace(full_name='root', gid=0, home='/root', name='root', shell='/bin/bash', uid=0))
```

and

```python
>> print(ret.account.full_name, ret.account.uid, ret.account.shell)
```

will output:
```text
root 0 /bin/bash
```

That's it! And the varlink python module is also very small.
Stay tuned for more and go read [Jens Reinmann's Java Varlink blog post](https://dentrassi.de/2017/12/17/varlink-for-java-what-wonderful-world-it-could-be/).