This Week in D June 14, 2015

Welcome to This Week in D! Each week, we'll summarize what's been going on in the D community and write brief advice columns to help you get the most out of the D Programming Language.

The D Programming Language is a general purpose programming language that offers modern convenience, modeling power, and native efficiency with a familiar C-style syntax.

This Week in D has an RSS feed.

This Week in D is edited by Adam D. Ruppe. Contact me with any questions, comments, or contributions.

Statistics

Major Changes

There was a big effort last week to clean up bugzilla, closing old D1 bugs as it is no longer supported. Summer cleanup on https://issues.dlang.org

A new version of the forum went live.

DConf 2015

DConf 2015 happened recently! Over 30 men gathered in person at Utah Valley University for about nine hours a day over three days to discuss D, with the majority of the conference also being livestreamed over Youtube to many other people.

The conference was also professionally recorded and those videos will be made available later, once editing is finished. This is underway now and they expect to start releasing them shortly.

Until then, the livestream has been chopped up on youtube by a user: DConf 2015: Individual talk videos from the livestream

This Week in D summarized the Wednesday morning session last week. This week, we'll continue our coverage.

Wednesday Morning Session

See May 31's issue.

Wednesday Afternoon Session

See last week's issue.

Thursday Morning Session

Day 2 of Dconf 2015 started off with Chuck Allison talking about his long work with C++ and how it brought him to D. Here are his slides.

Chuck started by showing some of his long history with C++, including books he has published, and mentioned that he was an early adopter of Zortech C++, written by Walter Bright, and on the C++ standards committee. Chuck also mentioned how Walter and Andrei Alexandrescu kept popping up in the C++ world over the years too.

One amusing anecdote he relayed was how Walter, at an early tech conference, gave a slideless talk using hand-written transparencies on an overhead projector.

Chuck also mentioned some of Java's successes Editor's note: Walter Bright was also in Java commercially for a while, writing a compiler for it too. and segued into D, mentioning how Walter gave a talk about it back in 2004 which intrigued Chuck.

Throughout his talk, Chuck used a website to ask for audience participation. The padlet urls in his slides are where people went to write very quick notes - more like pasting tags onto a canvas - to contribute some thoughts. One of these were asking what people like about D.

The summary of what Chuck likes about D is that it covers all the bread and butter needs conveniently with a number of language and library features. He also specifically called out fibers as coroutines as cool too, and commented that ranges are very cool. He called D a "nuclear reactor included" language, comparing to Python's "batteries included" slogan.

He ended with a list of D tools, again returning to padlet to get audience input, and praised D books.

The second talk was scheduled to be Martin Nowak about garbage collection, but due to a last-minute complication, he was unable to attend.

Instead, five other speakers volunteered to give lightning talks in his place.

Lightning talk #1 Editor's note: sorry, I forgot to write down who was speaking and I don't remember! talked about code coverage with dmd -cov. One trick he noted was dmd_coverDestPath("dir");, a magic function available if you declare the interface, could be called to change the path where the coverage report is written.

Lightning talk #2 was given by me, Adam D. Ruppe, and very briefly discussed This Week in D! My slides are available as HTML.

I talked about the process behind This Week in D - that is is mostly ad hoc, I use a ddoc template each week but collect the links by hand and write the commentary based on my experience being a frequent reader and contributer to the forums over about eight years now.

I boasted of myself and had slight criticism for the ddoc syntax, namely that $ is an unfortunate macro symbol because of how common it is in other usage, but hey, it is good enough.

Andrei mentioned that there's a DDOC_UNDEFINED_MACRO you can define to expand in cases where a macro is unknown, to avoid it being silently disappeared. I said the problem with that is that it is opt in and me, being a programmer, am lazy and tend to use defaults. It is worth using that in your macro file though to catch typos!

Finally, I asked people to contribute their content to me! My email address is in each issue of This Week in D, and you can catch me on IRC as either adam_d_ruppe or destructionator (depending on which client I'm using at the time) and I'm usually willing to chat about just about anything.

Lightning Talk #3 was given by Lionello Lunesu and mentioned value range propagation across statements and inside if statements, for example if(a < 256) { ubyte b = a; /* could work with cross-statement VRP */ }, and also signed/unsigned comparisons and the pitfalls they bring.

Lightning Talk #4 was Erik Smith discussing the outline of his std.database proposal, which you can also read about on the forum. He told me this weekend that he is making progress and expects to have more to show in another week or so - it has already integrated ODBC support and direct support for native drivers is coming soon.

Finally, Lightning Talk #5 was Walter Bright talking about doing stuff with the language as it is, focusing on the std.typecons.octal pattern, calling out the trick of converting an int to a string then back to an int to create a user-defined literal editor's note: Walter often credits me with conceiving this pattern, and while I wrote the main implementation and innovated the octal!int overload, the original idea was Andrei Alexandrescu's. He did credit Andrei in the spoken talk too, but not in the written article.

With the user-defined literal pattern, it is also possible to make new types like halffloat in the library with a fairly natural syntax - showing D's extensibility and flexibility.

I took the mic temporarily to mention that while std.conv.octal is a cool pattern, I rarely actually use it because it has too big of an import graph - std.conv brings in a lot of code for just an octal literal, which affects build times and binary sizes. This is something we can fix by writing more independent modules in Phobos, a known problem that is slowly being worked toward being solved (note for example std.algorithm and std.range being broken up into importable small chunks and larger implementation modules, using the package.d feature to maintain compatibility with code that expects to import it all).

Walter also mentioned dmd's code coverage again, saying it should be used with unittests to ensure they actually exercise the code.

Finally, before lunch, Mihails Strasuns (aka dicebot) gave a talk on porting Sociomantic's sizable D1 codebase to D2. Here are his slides.

He started by explaining that Sociomantic is an established and growing company that has depended on D since 2009 and described their requirements: that they wanted to move to D2, but not disrupt their daily development needs. This meant they wanted to switch to D2 while maintaining D1 compatibility, so they could fallback to the old compiler at any time.

They did this by making helper files that bridged the two languages together, with a lot of use of string mixins. The usage code would call these the same way in D1 and D2, then the implementation code encapsulates the differences. The biggest difficulty was with const Editor's note: transitive const/immutable was probably THE breaking change that really differentiated D2 when it was new. It came slightly after the version change, but it was the time when code seriously started to change. because before, it was shallow qualifier, and now, it is a deep change - const can only call other const, so changing it in one place at a time is not enough.

Dicebot explains how they made a d1tod2fix program, based on Brian Schott's libdparse, to help automate parts of the conversion. It did the mechanical conversions, like manifest constants from const to enum, in a 'good enough' manner; imperfect, but a huge help to them.

After language changes, they had to work on library changes, the most significant being the array stomping change in D2 - where resizing an array causes reallocation so a modified slice doesn't affect the original (maintaining the illusion that they are passed by value to functions). D1 made no attempt to prevent this array overwriting, and Sociomantic's code depended on that for their high performance code. They needed to keep the old behavior.

On the bright side, druntime does include a function: assumeSafeAppend, which activates the old behavior. Sociomantic thus had to call that when they needed to and were able to use a helper function to make it work in migration. Tedious and bug prone if they missed a spot, but they made it work.

The last major change they had to work with in the runtime library is to the garbage collector. Sociomantic uses a custom GC written by Leandro Lucarella which runs concurrently, avoiding pauses in their very time-sensitive code. They ported this to D2 but are planning to do a rewrite and more benchmarks to ensure it still fits their needs.

Dicebot now moved on to talking about language changes in general. He broke them up into three types:

Good language changes give a clear fix in the error message and don't change semantics. These give them a chance to fix it at their pace without much worry about silent breakage, while reducing their overall technical debt; making the code better in the long run.

Bad language changes have big semantic changes with no clear migration path; a big bunch of code has to be rewritten at once to make it work and the compiler error messages don't even help much. These are big cost and leave the user feeling abandoned by the language.

Finally, ugly changes are the silent ones that just pop up at runtime without any indication before then. It can bring performance problems or correctness problems and cannot be statically tested.

He summed it up by saying "please break my code" - a compile error is the nicest form of change, and if it yields a good return on investment for the user - it reduces technical debt or makes happier developers (because unhappy devs write unhappy apps!) while having an acceptable migration cost thanks to error messages and documentation guides - they want those changes. The short term cost is affordable and the long term returns are significant. Dicebot said that we shouldn't freeze the language and treat all potential changes as all bad without considering the other factors involved.

He also mentioned that deprecation periods are nice and dmd's version number is arbitrary and basically useless. We need better migration instructions with each new release. Jonathan Davis also took the mic to comment that people should please test their code with the dmd betas to catch regressions.

He finished the talk by saying Sociomantic is hiring D developers and PHP/Javascript developers in Berlin, Germany.

That led to lunch.

Thursday Afternoon Session

Coming back from lunch, the first speaker was Andy Smith, talking about using D in the world of financial software. Download his slides here.

He got interested in D because of Walter and Andrei's names and took a look. He said he wrote a bit of code that didn't work and asked for help on the #d IRC channel on freenode and was surprised to find that somebody on IRC actually meant it when he said he'd help rewrite the example so it works and explain it - Andy said this left him with a good impression of the community.

After this experience, he started using D at work. His company wanted to rewrite their infrastructure to take advantage of new computer hardware and he was going to use D for the prototype, but it ended up working well right into production.

D scores highly on all requirements they had: it was correct, testable, reliable, modifable, productive, and performant.

Among other wins in the language were: fast builds, easy testing, C syntax is easy learning, low level functions, the standard library, the module system helps with changes Editor's note: I'd also point out how much static typing helps in modifying code, and he found no nasty language surprises.

In Phobos, he loved having the modules for: getopt, json, DateTime, atomics, bitops, csv (he specifically called out how nice the Struct[] population functions are) and loved how well they worked and that they were easy to use.

Andy then moved on to talk about their system. Conceptually, it is a state machine with pure functions, which makes it auditable (very important in finance so regulators can check it), repeatable (helps debugging), and parallelizable (big win for performance).

Of course, this abstract model doesn't perform maximally well, so they use an implementation based on mutation for performance, but that is the general idea. Their code works in a couple layers: an inner layer that does business logic and an outer layer that handles concurrency and other matters. The outer layer is written in D.

See Andy's slides for more information he discussed about their architecture - explaining ring buffers, event sourcing, and more.

At this point, he explained how the original goal was to prototype it in D and rewrite in C++ for production, but D worked so well that they stuck with it. Among the wins he listed was:

Inline asm is available when it is needed, like for their lock-free ring buffer. Thanks to mixin templates, it was easy enough for mortals to use.

A padded template was written to avoid false sharing. This code was nicer in D than in Java. He noted that they used 128 bytes because of the prefetcher chunk size, larger than a cache line! There was a measured speed improvement. Editor's note: one nice thing about D's easy to use templates is that it is easy to tweak values like this and try measuring again to see what works best.

Reading data from java is a pain, needs unsafe module and offsets and such. D used compile time reflection and .offsetof, etc., properties to get rigth into the struct. With the D, we can generate the Java code.

C style string processing was easy in D too. They recycled a buffer and passed the buffer to an external API. Between extern(C) interaction and the familiar code, this was good for comfort as a newbie!

He summed it up by saying D was well worth it and faster than C.

The next speaker was Jonathan M. Davis giving an introduction to ranges. You can see his slides here. Editor's note: I know ranges well, so I didn't take extensive notes through everything he said - I encourage you to look through the slides too, he covered a lot of good material.

Jonathan started with an introduction to D's arrays and slices, explaining that their semantics are the same as a struct with a length and pointer member - slices are passed by value, but have a reference to the data.

His experience showed quickly as he warned us to beware of the implicit slicing of static arrays:

    char[4] staticArray; // not the same as a slice!
    char[] slice = staticArray; // but it will be implicitly sliced to one
    // slice now points to temporary data which can lead to crashes
    // or memory corruption without the type system warning you!

Jonathan then gave a good explanation of ranges, see his slides for more information. He notes that you need to import std.array; to treat built in arrays/slices as ranges, but warns us to remember that a slice is not a container - iterating over it with a range will consume the slice.

Since ranges aren't containers are are consumable views, they never grow and can only shrink. You can't append to a range. If you want to append, you need a container. You also cannot unpop a range, once you go forward with popFront (or popBack), the element is consumed.

Ranges have a lot of attributes like infiniteness which can be tested at compile time. However, the compiler will allow you to foreach over an infinite range (makes sense since you can break from it!), Jonathan warned to be careful with this since unless you do break, it will go forever!

In his slides, Jonathan presented an example random number range, showing how ranges move beyond iterating over a container and can also be used as generators. Andrei commented that his example wouldn't quite be uniform, but Jonathan said pretend it is for the sake of discussion.

Jonathan said that if you have a range problem, std.algorithm can probably solve it, but you might need to think of new names. For example, if you are looking for dropUntil, you won't find it... but std.algorithm.find actually does the same thing.

He then went on to talk about a big potential pitfall: passing a range to a function may copy it... which actually has undefined behavior! Since an input range may consume a nonrenewable resource, such as an underlying network stream, copying it might consume all views into it... or might not.

The range paradigm has a solution for this: a save function, provided by forward ranges. If this method is available, you can copy the range with expected results. If not, copying it is undefined. Editor's note: all range primitives are supposed to be low cost, if you are writing a range and need to allocate memory to make a copy, do not provide save - it is expected by algorithms to be a nearly free call and making it slow can break algorithm's performance guarantees!

Jonathan notes that Phobos often gets this wrong - you should call save when passing a range to equal for example, since otherwise you consume it while checking it.

He also talked about the string debate: Phobos treats strings as ranges of dchar rather than char in an attempt to be more Unicode correct. However, since this doesn't cover all graphemes - what the user perceives as a character - such as two Unicode characters that combine into one, for example, e followed by the add-accent character - Phobos' behavior still isn't really right. Bottom line: if you want your program to be unicode aware, you need to be aware of the underlying issues too, there is no magic bullet, despite Phobos' efforts.

Atila asked a question about walkLength: why use it instead of array? Jonathan answered that there's two advantages: first, walkLength won't actually walk if it doesn't have to - if the range provides a length member, it will forward to it. Secondly, walkLength is more efficient because it avoids allocating an array to hold the result when all you need is the length count.

Finally, Jonathan introduced output ranges, but noted that they aren't well defined and lazy input ranges - transformative things like std.algorithm.map - are often a better choice.

The next speaker was Mark Isaacson, a relative newbie to D, talking about his experience using it to reduce code smell in some C interaction. His slides are here. The recurring theme was the ability to define abstractions in D to make the code prettier and more correct, without sacrificing performance.

Mark explained that he had to write a ODBC driver which included some messy bits like writing a Windows DLL and being tied to some fairly ugly C function signatures. Early on, he raved about how nice it is to slice C strings to form D strings, demonstrating a short function that also leverages inout and isSomeChar to work across all kinds of strings in just a few simple lines. Similarly, he wrote a struct to handle output strings, which functions would write to.

Next, he talked about memory management in DLLs, which must pass data to and from C and cannot rely on the GC nor the operating system to clean it up properly. He wrote helper functions around malloc and free to manage all the memory manually with convenient syntax. He also called out the usefulness of core.sys.windows.windows in easily accessing the OS function MessageBoxW to pop up messages.

He also mentioned how dispatching variant types through foreach(type; TypeTuple!(list,here)) is easy to write and optimizes fairly well.

He then mentioned a few things that didn't work so well: he said dmd is too fast, rendering his makefile-fu obsolete. He also said the real type is broken on Windows This is due to an old ABI bug.., and also that dub didn't have a gui, so he had to call out to Notepad instead.

He then went back to the love, saying D rocks for scripting thanks to the fast dev cycle, the rdmd shebang support, the ease of use of std.process, and the fact that it has tested functions that can be reused, unlike bash which typically has poorly tested functions - if they have tests at all.

He also liked a memoize pattern: write a static nested function inside the visible function, and return a std.functional.memoize version of it. He also liked the UFCS pipeline chain style of functions:

stdin
  .byline
  .filter!(a=>keepLine(a))
  .map!
  ..... etc

One problem he had with this style was debugging, so he wrote a debugRange which prints what passes through it and keeps going. Note that std.range.tee can do this too. Someone commented that range.front ought to be idempotent; it should do nothing beyond returning the value, as it is often called multiple times and extra behavior is not expected.

One thing Mark mentioned about debugging is he likes to stomp values with all zeroes so their reuse is more obvious in a debugger. Walter commented that using some value other than zero might make it even easier to notice.

Mark summarized by saying that D does well in building safe abstractions on top of smelly code, works well for scripting, and has a few rough edges (mainly due to DLL work), but generally works well.

We finished out Thursday with a hot mic session, which lasted over an hour. It was originally meant to be focused on the release process, but it ended up being far more random, with a lot of people discussing a lot of different things.

What follows is my notes on what was discussed, roughly formatted. It is NOT a transcript and may have errors or biases introduced by me.

Andrei says 500k visits in March, up from 1,500 in 2010. 1,000 dmd downloads per month. Andrei says github activity lags the rest of the growth.

Liran: when we use D, people ask, why not Rust or Go? D needs PR. Andrei: we need to be the best we can without fighting anyone. Rust has good things going for it but challenges too. D needs a big backer. Walter: often someone blames the compiler or says what it doesn't do right and doesn't talk enough about what it does right. Worth remembering what we are doing right and little things aren't dealbreakers. Joseph: interesting psychological thing where the more you get right the more they see the wrong stuff.

Joseph: with the foundation is there a plan to find wider consensus? Andrei: no. what we hope with the foundation is financing things like conference or pro support. I like the idea of attaching a community process though.

Online: when do we plan to have an ANSI standard? Walter: not now, it might come later but no real plans. Chuck: we need vested users to justify the slow moving standardization process. We should not go that way until we are compelled by it. Andrei: we want to stabilize before standardize, not standardize to become stable. It will take years.

Atila: shared finalization coming? Andrei: everything needs to come. Walter: an advantage of standardization is forcing us to formalize it. Andrei: Brian's work is good for frontend, we need something similar for semantic. Chuck: we should be careful where we air the dirty laundry, C++ committee divided into working groups and that's where biggest problems are discussed and they work on it and present solution to publicize. Andrei: I plan to start a blog and talk about good stuff and bad stuff...

Q: lot of code takes predicates. Params can create closures and GC it. Hurts codegen. What will we do? Walter: we would like to resolve it with ownership semantics, e.g. map promises not to escape so no closure is needed. More annotations will come... Brad: it does escape in the return value! Andrei: we need an optimizer which inlines and optimizes aggressively.

Online: planning on updating TDPL or writing a new one? Andrei: a second edition may be coming, the publisher is interested in more D stuff. Far off, I might write another book. So it is on the list. Chuck: there is a video course on D on pluralsight. It's not bad. Walter: a roku or youtube channel would be interesting with D videos.

Andrei: we need more talent. OK work is the enemy of great work, we need to raise the bar. Great work is instantly attractive. John (I think): wider library is important, a lot of OK is worth it. Andrei: if there is no work, OK work is great work! Amaury: we need to focus on deeper stuff not just names and junk. Walter: we should ask why pull it, not why not. David: we need ok to be a stepping stone to great... don't turn people away because it isn't great. Andrei: i try to make weak proposals better. Amaury: fixing little things is great for newcomers, big bang for buck. We need a guide for newcomers. Andrei: yes we need stuff for beginning contribs. Documentation needs work. I'll take ANY work on it. Walter: I agree, just adding examples to Phobos would be great.

Leonello: how do we decide what to do? Walter: we could just pick something at random and fix problems we find! Not just examples but maybe fun examples. Moving stuff from SO might be a good idea. Waler: just one step closer on bug reports helps too. Brian: documenting return values is valuable. dicebot: mediocre PRs often are blockers on my walk to great work. Walter: these aren't absolute rules. B-29 Bomber!!!! They duplicated damage too while following orders too literally. Andrei: these are judgement calls. Amaury: just good enough can sometimes discourage meta. comment: yay for std.meta, it was too complex to explain typetuples and such. dicebot: comment Andrei: typetuple was a liability we owe you an apology for not being decisive enough.

Brian: the longer we wait to fix parser bugs the more stuff that needs fixed. What's the plan? Walter: we need to look at one at a time. Brian: the throwable one is both-ways compatible, is that ok? Walter: is there any correct existing code that would break for this? Brian: are we committing to error stability and warning stability, that even new bad broken code won't be warned? Walter: case by case. If the code is reasonably ok it shouldn't pull the rug out.

Brad: when is allocator coming? Andrei: std.experimental.allocator is ready for review next week.

Friday Sessions

The report on the final day of DConf must be delayed one last week as it is not ready to go at this publication time, sorry for the wait!

Open D Jobs

A new page has been added to the D Wiki listing open D jobs. Take a look if you're interested, and add yours if you know of one that is available!

In the community

Community announcements

See more at digitalmars.D.announce.

Significant Forum Discussions

See more at forum.dlang.org and keep up with community blogs at Planet D.

Learn more about D

To learn more about D and what's happening in D: