November 09, 2015

Rust impressions from a C++/D programmer, part 1

Discussion on programming reddit

Discussion on Rust reddit

C++ and D aren’t the only languages I know, I labeled myself that way in the title because as far as learning Rust is concerned, I figured they would be the most relevant in terms of the audience knowing where I’m coming from.

Since two years ago, my go-to task for learning a new programming language is to implement an MQTT broker in it. It was actually my 3rd project in D, but my first in Haskell and now that I have some time on my hands, it’s what I’m using to learn Rust. I started last week and have worked on it for about 3 days. As expected, writing an MQTT broker is a great source of insight into how a language really is. You know, the post-lovey-dovey phase. It’s like moving in together straight away instead of the first-date-like “here’s how you write a Scheme interpreter”.

I haven’t finished the project yet, I’m probably somewhere around the 75% mark, which isn’t too shabby for 3 days of work. Here are my impressions so far:

The good

The borrow checker. Not surprising since this is basically the whole point of the language. It’s interesting how much insight it gave me in how broken the code I’m writing elsewhere might be.  This will be something I can use when I write in other systems languages, like how learning Haskell makes you wary of doing IO.

Cargo. Setting up, getting started, using someone’s code and unit testing what you write as you go along is painless and just works. Tests in parallel by default? Yes, please. I wonder where I’ve seen that before…

Traits. Is there another language other than D and Rust that make it this easy to use compile-time polymorphism? If there is, please let me know which one. Rust has an advantage here: as in Dylan (or so I read), the same trait can be used for runtime polymorphism.

Warnings. On by default, and I only had to install flycheck-rust in Emacs for syntax highlighting to just work. Good stuff.

Productivity. This was surprising, given the borrow checker’s infamy. It _does_ take a while to get my code to compile, but overall I’ve been able to get a good amound done with not that much time, given these are the first lines of Rust I’ve ever written.

Algebraic types and pattern matching. Even though I didn’t use the former.

Slices. Non-allocating views into data? Yes, please. Made the D programmer in me feel right at home.

Immutable by default. Need I say more?

Debugging. rust-gdb makes printing out values easy. I couldn’t figure out how to break on certain functions though, so I had to use the source file and line number instead.

No need to close a socket due to RAII. This was nice and even caught a bug for me. The reason being that I expected my socket to close because it was dropped, but my test failed. When I looked into it, the reference count was larger than 1 because I’d forgotten to remove the client’s subscriptions. The ref count was 0, the socket was dropped and closed, and the test passed. Nice.

No parens for match, if, for, …

The bad

The syntax. How many times can one write an ampersand in one’s source code? You’ll break new records. Speaking of which…

Explicit borrows. I really dislike the fact that I have to tell the compiler that I’m the function I’m calling is borrowing a parameter when the function signature itself only takes borrows. It won’t compile otherwise (which is good), but… since I can’t get it wrong what’s the point of having to express intent? In C++:

void fun(Widget& w);
auto w = Widget();
fun(w); //NOT fun(&w) as in Rust

In Rust:

fn fun(w: &mut Widget);
let w = Widget::new();
fun(&mut w); //fun(w) doesn't compile but I still need to spell out &mut. Sigh.

Display vs Debug. Printing out integers and strings with {} is fine, but try and do that with a Vec or HashMap and you have to use the weird {:?}. I kept getting the order of the two symbols wrong as well. It’s silly. Even the documentation for HashMap loops over each entry and prints them out individually. Ugh.

Having to rethink my code. More than once I had to find a different way to do the thing I wanted to do. 100% of the time it was because of the borrow checker. Maybe I couldn’t figure out the magical incantation that would get my code to compile, but in one case I went from “return a reference to an internal object, then call methods on it” to “find object and call method here right now”. Why? So I wouldn’t have to borrow it mutably twice. Because the compiler won’t let me. My code isn’t any safer and it was just annoying.

Rc<RefCell<T>> and Arc<Mutex<T>>. Besides the obvious “‘Nuff said”, why do I have to explicitly call .clone on Rc? It’s harder to use than std::shared_ptr.

Slices. Writing functions that slices and passing them vectors works well enough. I got tired of writing &var[..] though. Maybe I’m doing something wrong. Coming from D I wanted to avoid vectors and just slice arrays instead. Maybe that’s not Rusty. What about appending together some values to pass into a test? No Add impl for Vecs, so it’s massive pain. Sigh.

Statements vs Expressions. I haven’t yet made the mistake of forgetting/adding a semicolon, but I can see it happening.

No function overloading.

Serialization. There’s no way to do it well without reflection, and Rust is lacking here. I just did everything by hand, which was incredibly annoying. I’m spoiled though, in D I wrote what I think is a really good serialization library. Good in the lazy sense, I pretty much never have to write custom serialization code.

The ugly

Hashmaps. The language has operator overloading, but HashMap doesn’t use it. So it’s a very Java-like map.insert(key, value). If you want to create a HashMap with a literal… you can’t. There’s no equivalent macro to vec. You could write your own, but come on, this is a basic type from the standard library that will get used a lot. Even C++ does better!

Networking / concurrent IO. So I took a look at what my options were, and as far as my googling took me, it was to use native threads or a library called mio. mio’s API was… not the easiest to use so I punted and did what is the Rust standard library way of writing a server and used threads instead. I was sure I’d have performance problems down the road but it was something to worry about later. I went on writing my code, TDDed an implementation of a broker that wasn’t connected to the outside world and everything. At one point I realised that holding on to a mutable reference for subscribers wasn’t going to work so I used Rc<RefCell<Subscriber>> instead. It compiled, my tests passed, and all was good in the world. Then I tried actually using the broker from my threaded server. Since it’s not safe to use Rc<RefCell<>> in threads, this failed to compile. “Good!”, I thought, I changed Rc to Arc and RefCell to Mutex. Compile, run, …. deadlock. Oops. I had to learn mio after all. It wasn’t as bad as boost::asio but it wasn’t too far away either.

Comparing objects for identity. I just wanted to compare pointers. It was not fun. I had to write this:

fn is_same<T>(lhs: &T, rhs: &T) -> bool {
    lhs as *const T == rhs as *const T;
fn is_same_subscriber<T: Subscriber>(lhs: Rc<RefCell<T>>, rhs: Rc<RefCell<T>>) -> bool {
    is_same(&*lhs.borrow, &*rhs.borrow());



I thought I’d like Rust more than I actually do at this point. I’m glad I’m taking the time to learn it, but I’m not sure how likely I’ll choose to use it for any future project. Currently the only real advantage it has for me over D is that it has no runtime and could more easily be used on bare metal projects. But I’m unlikely to do any of those anytime soon.

I never thought I’d say this a few years back but…I like being able to fall back on a mark-and-sweep GC. I don’t have to use it in D either, so if it ever becomes a performance or latency problem I know how to deal with it. It seems to me to be far easier than getting the borrow checker to agree with me or having to change how I want to write my code.

We’ll see, I guess. Optimising the Rust implementation to be competitive with the D and Java ones is likely to be interesting.

November 06, 2015

Writing a vibe.d app

In this tutorial we’ll write a vibe.d web app that links bugzilla issues to github pull requests.

Let’s start by creating a vibe.d project,

dub init dlang-bot --type=vibe.d
git init dlang-bot

which generates a few files and folders.

dub.sdl      # the dub package file
source/app.d # a hello world vibe.d app
public       # folder for assets
views        # folder for frontend templates

Let’s edit app.d and use vibe.d’s URLRouter to add an endpoint for github webhooks, so that the app can receive notifications about new or updated pull requests.

auto router = new URLRouter;
    .get("/", (req, res) => res.redirect(""))
    .post("/github_hook", &githubHook)
listenHTTP(settings, router);
void githubHook(HTTPServerRequest req, HTTPServerResponse res)
    if (req.headers["X-Github-Event"] == "ping")
        return res.writeBody("pong");
    assert(req.headers["X-GitHub-Event"] == "pull_request");

    // vibe.d parses application/json post bodies by default
    auto action = req.json["action"].get!string;
    logDebug("#%s %s", req.json["number"], action);
    switch (action)
    case "opened", "closed", "synchronize":
        auto commitsURL = req.json["pull_request"]["commits_url"].get!string;
        auto commentsURL = req.json["pull_request"]["comments_url"].get!string;
        runTask(toDelegate(&handlePR), action, commitsURL, commentsURL);
        return res.writeBody("handled");
        return res.writeBody("ignored");

The hook can handle ping and pull_request events and will process interesting pull request actions asynchronously in a separate task.

void handlePR(string action, string commitsURL, string commentsURL)
    auto comment = getBotComment(commentsURL);
    auto refs = getIssueRefs(commitsURL);
    logDebug("%s", refs);
    if (refs.empty)
        if (comment.url.length) // delete any existing comment
    auto descs = getDescriptions(refs);
    logDebug("%s", descs);
    assert(!(r =>!(d =>;

    auto msg = formatComment(refs, descs);
    logDebug("%s", msg);

    if (msg != comment.body_)
        updateBotComment(commentsURL, comment.url, msg);

The processing is fairly straightforward. First we get a list of issues references from any of the PR’s commit messages, then ask Bugzilla for a short description of those issues, format the information, and finally create, update, or delete a comment on the PR.

struct IssueRef { int id; bool fixed; }
// get all issues mentioned in a commit
IssueRef[] getIssueRefs(string commitsURL)
    // see
    enum issueRE = ctRegex!(`((close|fix|address)e?(s|d)? )?(ticket|bug|tracker item|issue)s?:? *([\d ,\+&#and]+)`, "i");
    static auto matchToRefs(M)(M m)
        auto closed = !m.captures[1].empty;
        return m.captures[5].splitter(ctRegex!`[^\d]+`)
            .map!(id => IssueRef(!int, closed));

    return requestHTTP(commitsURL, (scope req) { req.headers["Authorization"] = githubAuth; })
        .map!(c => c["commit"]["message"].get!string.matchAll(issueRE).map!matchToRefs.joiner)
        .sort!((a, b) => <

This code heavily uses std.algorithm and std.range for pipeline style operations. It’s pretty terse but much more robust (and simple) than writing explicit nested loops. The pipeline fetches all commits, matches all referenced issues in commit messages, converts the matches, joins all references, joins all references of all commits, and sorts them by issue id.

struct Issue { int id; string desc; }
// get pairs of (issue number, short descriptions) from bugzilla
Issue[] getDescriptions(R)(R issueRefs)
    import std.csv;

    return ",%)&ctype=csv&columnlist=short_desc"
        .format(!(r =>
        .sort!((a, b) => <

The code to query Bugzilla is fairly similar but uses csv instead of json.

struct Comment { string url, body_; }

Comment getBotComment(string commentsURL)
    auto res = requestHTTP(commentsURL, (scope req) { req.headers["Authorization"] = githubAuth; })
        .find!(c => c["user"]["login"] == "dlang-bot");
    if (res.length)
        return deserializeJson!Comment(res[0]);
    return Comment();

I’ll spare the formatting code, you can find it in the full source code.

Getting an existing comment simply searches for the first comment posted by the dedicated dlang-bot user and uses deserializeJson to convert json to a struct.

void sendRequest(T...)(HTTPMethod method, string url, T arg)
    if (T.length <= 1)
    requestHTTP(url, (scope req) {
        req.headers["Authorization"] = githubAuth;
        req.method = method;
        static if (T.length)
    }, (scope res) {
        if (res.statusCode / 100 == 2)
            logInfo("%s %s, %s\n", method, url, res.bodyReader.empty ?
                    res.statusPhrase : res.readJson["html_url"].get!string);
            logWarn("%s %s failed;  %s %s.\n%s", method, url,
                res.statusPhrase, res.statusCode, res.bodyReader.readAllUTF8);

void deleteBotComment(string commentURL)
    sendRequest(HTTPMethod.DELETE, commentURL);

void updateBotComment(string commentsURL, string commentURL, string msg)
    if (commentURL.length)
        sendRequest(HTTPMethod.PATCH, commentURL, ["body" : msg]);
        sendRequest(HTTPMethod.POST, commentsURL, ["body" : msg]);

And eventually we need some code to create, update, and delete comments. Notice that requestHTTP is synchronous and uses callbacks only to prevent you from escaping the request/response object.

Now that we’ve implemented our app and tested it locally (using ngrok for example) we can deploy it on Heroku. Though first we have to bind to an external network interface and make the listening port configurable.

auto settings = new HTTPServerSettings;
settings.port = 8080;
settings.bindAddresses = [""];
readOption("port|p", &settings.port, "Sets the port used for serving.");

Unfortunately openssl on heroku is configured to use custom config and certificate paths, so we have to explicitly specify Ubuntu’s default CA bundle.

// workaround for openssl.conf on Heroku
HTTPClient.setTLSSetupCallback((ctx) {

After that we use the Heroku CLI to create a new app, push+deploy our code, set our github token, and start a single dyno to serve the app.

heroku create dlang-bot --buildpack
git push heroku master
heroku config:set GH_TOKEN=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
heroku ps:scale web=1

November 02, 2015

My first D Improvement Proposal

After over a year of thinking about this (I remember bringing this up at DConf 2014), I finally wrote a DIP about introducing what I call “static inheritance” to D.

The principle is similar to C++ concepts: D already has a way of requiring a certain compile-time interface for templates to be instantiated, the most common being isInputRange. What is lacking right now in my opinion is helpful compiler error messages for when a type was intended to be, e.g. an input range but isn’t due to programmer error.

I tried a library solution first, assuming this would be easier to get accepted than a language change. Since there was nearly 0 interest, I had to write a DIP. Here’s hoping it’s more succesful.

November 01, 2015

General Update

I want to use this way to announce some news that I did not have the time to blog about. These news divide esentually in two categories: career and projects update:

Career Update:

Since August 2015 I am back to the games industry! After an intermezzo as a consultant I am now a Team Lead Software Developer at InnoGames in Hamburg, Germany. I love to be back in this vibrant and challenging branche.

I am lucky and have the opportunity to lead my team from the ground up in a whole new and exciting project. If you are an experienced Unity3D Developer or Java Backend Developer and you would like to work with me please drop me a line or find infos on these jobs here: InnoGames Careers

Project Updates:

There are also some new open source efforts that maybe of interest:

bgfxd-examples: bgfx examples ported to D

I started to like bgfx a lot and since there are D bindings I tried to learn using it by porting the official examples to D. I did not finish a lot yet but it is an ongoing process.


file2d: create embedable files in/for D

This little helper was designed to create embedable binary files for D. You can pass in any file and it will be read as binary and converted to a module containing a symbol that contains this files content as a hex string like this:

static immutable ubyte[1234] = [0x12, ...];

FMOD D Bindings

I created and maintain FMOD bindings for the D programming language for a couple of months now but I only recently added a simple but illustrative application on how to actually use it in D. See the example app here on github

Actually it uses a freeware sound converted to compile time string using file2d mentioned above (and it is this usecase for which i created file2d)

Steamworks D Bindings

I also created Steamworks SDK bindings a couple of months back already. The first version of the SDK that was supported is 1.34 since it was the first that included the c-api that I am binding. The Bindings include a simple demo app too to demonstrate on how to use the API.

October 30, 2015

DerelictGL3 Update

The reload method of the DerelictGL3 and DerelictGL loaders now supports two GLVersion parameters.

The first represents the minimum required version of OpenGL that a context should support. Previously, it was necessary to check the return value of reload to determine if you got a high enough version. Now, when a version other than GLVersion.None is specified, the loader will throw an exception before attempting to load anything if the currently active context does not support that version. The default value, GLVersion.None, causes the function to behave as it always has.

The second parameter allows you to constrain the loader to a maximum version. This will force the loader to load no higher than the version specified by this argument, even if the context supports a higher version. The default value is GLVersion.HighestSupported.

As an example, the following will load any version up to and including OpenGL version 4.4.

DerelictGL3.reload(GLVersion.None, GLVersion.GL44);

Generally, the first parameter is more interesting. For example, if your app requires OpenGL 3.3 or higher, you can pass GLVersion.GL33 as the first argument. The second parameter is useful for rare and specific circumstances, such as avoiding driver bugs. For example, one DerelictGL3 user has a driver that reports support for GL 4.5, but some of the 4.5 functions repeatedly fail to load with that driver. Assuming your app requires no 4.5 functions, the second argument will allow the app to load even when that buggy driver is present.

October 29, 2015

Worth Reading (October 2015)

The following stuff is what caught my eye in the last couple of weeks. Consider these kind of posts as an entry in my personal knowledge base ;)

Book: “Endurance: Shackleton’s Incredible Voyage” link

I found this book by Alfred Lansing absolutely stunning. Both a great adventure and a documentary about one of the most spectacular expeditions in modern history. It tells the story of an trans-antartic expedition turned into an breathtaking adventure of an inspiring leader trying to save the lives of his crew. This story is not only tense and freaking real it is an extraordinary example of inspiring leadership!

Post: “SpringSource Certified Spring Professional” by Jakub Staš

Blog Post

I was looking for information on a Spring Certification and stumbled upon this very interesting blog post from a guy who took this certification too.

Post: “Using Google Protocol Buffers with Spring MVC-based REST Services” Post

In this post it is described how to make Spring able to return webrequest results in protobuf format automatically.

Note: Maybe worth mentioning to be sure to add maven project “protobuf-java-format” into the dependencies to be actually able to convert proto results to json etc.

Sound Library: “SoLoud” by Jari Komppa

Thanks to a twitter tweet (that I can’t seem to find again) I stumbled upon this nice little sound library: SoLoud

It is crossplatform and feature most of the features that an indie dev needs and more. I was always looking for a really free alternative to fmod and will definitely look into this as soon as I find the time. Especially since it features a dlang binding right off the shelve: SoLoud D-API.

Post: “es_core: an experimental framework for low latency, high fps multiplayer games” by TTimo

Blog Post

TTimo talks about an interesting multiplayer game engine design specificly with low latency FPS multiplayer games in mind. He uses a highly concurrent design with no data sharing but message passing only while avoiding copys and “all memory heap operations”. I like this design, I maybe adopt some of it in unecht once I finally have some time to work on that again!