This Week in D June 21, 2015

Welcome to This Week in D! Each week, we'll summarize what's been going on in the D community and write brief advice columns to help you get the most out of the D Programming Language.

The D Programming Language is a general purpose programming language that offers modern convenience, modeling power, and native efficiency with a familiar C-style syntax.

This Week in D has an RSS feed.

This Week in D is edited by Adam D. Ruppe. Contact me with any questions, comments, or contributions.

Statistics

DConf 2015

DConf 2015 happened recently! Over 30 men gathered in person at Utah Valley University for about nine hours a day over three days to discuss D, with the majority of the conference also being livestreamed over Youtube to many other people.

The conference was also professionally recorded and those videos will be made available later, once editing is finished. This is underway now and they expect to start releasing them shortly.

Until then, the livestream has been chopped up on youtube by a user: DConf 2015: Individual talk videos from the livestream

This Week in D summarized the Wednesday morning session last week. This week, we'll continue our coverage.

Wednesday Morning Session

See May 31's issue.

Wednesday Afternoon Session

See June 7's issue.

Thursday Sessions

See last week's issue.

Friday Morning Session

DConf 2015 was a three day event, ending on Friday after six more talks. Friday morning kicked off with Andrei Alexandrescu delivering his mysterious sounding talk labeled Generic Programming Must Go. See his slides here. The talk was mostly prompted by his work on std.allocator.

He started by defining generic programming with its pros and cons, saying it is obsessed with naming things. In generic programming, you target generic concepts. Editor's note: this is similar to writing functions taking an interface in Java style object orientation in principle (but different in that the generic programming model statically dispatches, so there's no indirection.)

These concepts have rigid definitions and can explode in names quickly if you try to cover all the variations. Andrei pointed out that ranges tried to be generic - defining InputRange, etc., but quickly needed more fine-grained control, so hasLength, isInfinite, and other variants were added, and it was very good - these can be tested individually rather than trying to define InputRangeWithLength and InfiniteInputRange and all the other variations it implies.

He then tied this back into allocators, explaining that they are "high vocabulary"; that is, they have several independent items that can vary, such as alignment, deallocation, reallocation, and more. Working a concept name onto all of these was not practical.

"Let's go Descartes", he said, meaning to throw everything out and try to reason from nothing but first principles, alluding to Descartes' starting point of assuming nothing exists but himself, who must, because if he didn't exist, he could not have this thought in the first place. Andrei then asked what the simplest allocator that could possibly work was, and started rebuilding the idea from there.

The simplest allocator just allocates (and has an alignment guarantee, even if it is just 1, working on the byte level). It can to a lot more, but that's all it strictly needs. Every other member is optional, and instead of trying to work them all into some kind of generic programming concept or OO tree, Andrei proposes design-by-introspection. That is, code that builds upon the allocators or uses them just checks for the presence of the member and adapts to whether it is there or not.

Thus, a level of maximum genericness is achieved and allocators with different capabilities can be combined by looking at their individual components with very little code. Andrei demonstrated this by showing a simple region allocator (returning slices into a stack buffer) and malloc based allocator, then combining them with a fallback allocator.

The fallback allocator, by looking for only the minimum methods it needs, can adapt to the best combination of input allocators (or issue a compile error if you ask it to do the impossible).

In usage code, if an allocator doesn't have a deallocate method, you don't have to think about it. If it does, you need to call it. This is checked with introspection - can I do this?

Andrei showed more code in his slides to demo the idea, how small pieces combine and static if makes the code simple. One thing he said verbally was that the code checks allocate.length rather than null because allocate(0) may legitimately return an empty array rather than null. Checking for the matching length is the simplest thing that works in all cases. An audience member asked about what happens if the allocator over-allocates, and Andrei said if it does, it should return a slice sized to what the caller asked for, so the length will match anyway upon success.

Andrei also said the hasMember in his example should check the signature too, to ensure they take the right arguments. Editor's note: I disagree, unless this is done carefully, because checking arguments too can mean a capability gets silently ignored on minor things like a const mismatch. But, we can do this correctly with a bit of library code: if a member exists but has a wrong signature, check that and static assert to reject it. If a member does not exist at all, ignore it, assuming it is intentionally left out because it is optional. That way, we get the introspection win while keeping the best available static checks. Andrei said he agrees that is a good approach when I talked to him about it after the talk.

Bottom line of the talk: concepts and traditional generic programming fail because they explode with combinations on lots of inputs. Static introspection with boolean constraints in static if win in this case - thus, design by introspection is Andrei's killer new approach to generic programming.

After Andrei (who thankfully went over his time, taking the pressure off short-on-material me :-D ) finished, Adam Ruppe (yea, even this editor!) went to the lectern talk a little on implementing a dynamic type in D. You can see my slides here as a PDF or here as HTML.

My key point was to say that the same D language I talked about last year for doing low-level kernel code can also write dynamic style code, and I showed some of the language features that can be combined to make a var type.

I started with a very basic background on what typing is: the idea that memory just holds numbers, and those numbers have some kind of meaning, some interpretation done by the program.

I then tried to roughly break up the myriad of typing systems into two axes: compile time checks vs runtime tags and implicitly coerced vs mismatches-as-error types, putting a few languages into the grid, verbally saying how this is a really fuzzy thing because different techniques may be available in the same language and not all techniques fit on these axes.

Throughout the talk, I had a blasphemously religious theme. In that vein at this point, I quoted from the Bible, James 1:5. The picture in the slide is of James Gosling, just a silly joke because his name is James. I said Walter prayed to the great computer for an answer to which language he should work on, and he received an answer where he saw the great computer appear before him and tell him to work on none of them, and instead write D.

Walter took the mic to claim my picture of his vision is a photoshop and he could tell because of the pixels. I told him he was wrong, it was a Microsoft Paint!

I then explained that D has a superior typing model, with static types, strong checks, convenience and power through inference and templates, and static assert for additional custom checks, all at compile time, and that the system improves each year.

Next, I started about doing the opposite of that: implicitly coerced, run time tagged typing in D:

// this is valid D code! 
var a = 10; 
var b = "20"; 
var c = a + b; 
var d = json!q{ "foo": { "bar": 10.2 } }; 
writeln(d.foo); // {"bar":10.2} 
d.foo.bar = (var a) => a ~ b; 
writeln(d.foo.bar()("hello! "));

Of course, this forces us to ask the question: what does D need with a dynamic type? I gave a few ideas: interacting with external APIs, working with binary-stable runtime interfaces, prototyping code, script interaction, and of course, showing off just because we can. The techniques used here can be used in whole or in part for a lot of other programming challenges we can solve with D.

I explained the basic technique: a tagged union, and explained how std.variant and my arsd.jsvar use variations on it, then went into the syntax sugar D has to make this easily usable: operator overloading and reflection, primarily.

I showed two main troubles I had: one was returning a delegate in a struct that uses local variables and struct variables. To which does the context pointer point, the struct or the stack? To solve this, you can be explicit: make a static nested function that accesses everything it needs through local variables. The other problem was returning a ref to a null: for this, I made a new var(null) and returned it. That generates a lot of garbage, but also works in all cases without needing to write a lot of special cased code.

Another technique I showed was looping over a TypeTuple to generate code for a lot of types, by mixing in each bit of code inside a runtime if. In the slide, I called it CtList instead of TypeTuple, but that's the same thing.

I also talked about my wish list of language features: I want implicit construction added (though explicit, unlike C++ - an implicit constructor should be marked with the keyword). I believe such a feature ought to be used sparingly, but is useful in some cases, especially when you want to write a struct to completely replace a built in type - without this, null cannot be implicitly cast to a user type.

I also mentioned implicit casts back to a static type, but shot it down because I think an explicit cast is better - with type inference, the code is concise anyway.

Finally, I said I really want @property to be fixed just to handle the getter returning callable edge case. I don't care about the other issues people have with it on the forums, this one case just needs to be fixed because code is wrong without it.

To wrap up, I showed how the dynamic type can easily wrap static types thanks to reflection then expose it to a script engine in just a few lines of code, showing off my half-finished, but basically working in 350 lines of code api inspector.

After the talk, I was asked if I made all the Mormon references just because we were in Utah. The answer is no, it was not just because of location: I'm actually a real-life Latter-day Saint, though I can be a bit irreverent at times :).

The third talk of the day was Joseph Wakeling talking about random number generation. His slides are online here.

Joseph discussed some random number theory and compared-and-contrasted D's approach with C++'s, then went right into the difficulty with thinking of pseudo random numbers as ranges: the algorithm is a good match for a forward range, but the semantics are a bit strange: the pass-by-value problem Jonathan discussed in his talk strikes again. It means the seed will be saved and not updated at the top level, which means the same number sequence will occur again in subsequent calls.

He pointed out that something similar can happen in C++.

He then started discussing solutions: we could reseed the generator each time, but that's verbose. We could also make random a new type of range: one that can save, but generally shouldn't; an input range with forward range capabilities but with a different name, perhaps dup rather than save so it is never done by algorithms which think saving is generally good.

Joseph explained that his future direction is to make random a reference range, try to separate the engine from the distribution better, and to implement more distributions like the options C++ offers.

In the question section, Dicebot asked: why would you want to save a random number state? Joseph answered you might save the seed when reloading games to help prevent cheating (reloading over and over until you get a better random result), or perhaps to rerun numerical simulations with the same input so you can debug it better. Editor's note: I used to save seeds with games along with timed user input to easily save replay files too. Same input, same algorithm, same output - a small file could store a whole game session that can be watched again for identical results each time.

Dicebot asked: have you considered generator vs range like container vs range? Joseph answered he had considered it a little, but not yet in depth; it is an interesting idea.

Steven asked: along those same lines, the nice range interface means people want to use them, so maybe we should change it so it doesn't fit the range and do the wrong thing. He said you can also do &generator and get a range too. Joseph said: that will sometimes work but not always, you probably want reference. Steven: the hard part is make it save because taking the address of a local can easily lead to escaping an invalidated reference.

We then broke for lunch.

Friday Afternoon Session

We returned from lunch to hear John Colvin, a physics grad student, speak about scientific programming in D. Here are his slides.

He started by explaining that scientific programming is a big area with different fields like simulations, data analysis, and more, and that it is done on a variety of hardware, from low-power microcontrollers up to gigantic supercomputers. It may also have some unique requirements like dealing with custom data formats. John described some of the requirements and status quo for simulations and data analysis in scientific programming, see his slides for a list.

John pointed out that many scientific programming is done by scientists who do not consider themselves to be programmers. Even if they spend most their day writing code, programming is not their speciality and they just want to get it done so they can focus more on the science they are really trying to do. As such, they are most interested in easily getting stuff done in a familiar manner to them and would benefit greatly from more task-focused, newbie-friendly tutorials.

A problem he pointed out is that as D enthusiasts, our web search results are likely personalized based on our history and give better results to us than it would someone else. He fired up the TOR browser to demonstrated the poorer results and emphasized that having tutorials would be important also to tell people *where* to look to a particular functionality.

He also called out the unimplemented array operations (some are implemented, many aren't) as a stumbling block. Array operations look great and familiar, and of course, useful to scientists, but when they aren't implemented it gives them trouble. He also called out the buffer reuse of byLine as being weird for newbies.

John then weighed in on D's pros and cons for two general areas of scientific programming: simulations and data analysis.

For simulations, he said: D wins because its multidimensional arrays are good enough, performance is excellent, it has good Linux support, foot-shooting protection, and a lot of flexibility. He said it needs better MPI support - a message passing interface used heavily in academia and could use easy libraries. He also said D lacks familiarity and a proven track record to scientists.

For data analysis, he said D's performance is a big win, as is its Linux support and flexibility, and its ability to interoperate with other languages and Matlab is good. For Matlab, he pointed out that the C api is pretty easy to use. He said D needs better OS X support, more shared library love, more focus on the GDC compiler for its excellent optimizing capabilities, and some tutorials for working with Matlab, so new people can easily get started with it. He said D lacks a standard multidimensional array with all kinds of slicing, a REPL and something like Python's Notebook integration, and familiarity. The standard multidimensional array is important so other people can write libraries that work together, like the Python community has, and he thinks the repl part might be usable with Python bindings.

He attacked D's real type, the 80-bit float, saying it is slow and generally unnecessary. He also said the GC can easily break the benefits of parallel code because of the global lock: slapping parallel on a loop can give big speedups fairly easily if you do everything right, but if you use the GC, it might lock all the threads regularly, negating the multi-core boost. Currently, it is just something you need to be careful about to use correctly for best performance.

John said a lot of scientific work works well on GPUs, but the barrier to entry is a bit high. He's working on a library to give D great GPGPU support, using OpenCL.

Finally, he pointed out the which he is running to try to get a variety of tested, vetted, and useful libraries together in an easy to find place. He asked for people's help in getting it better.

Our penultimate speaker was Atila Neves, talking about testing D with Cucumber. His slides are here.

He started by talking about what tests are: unit tests ought to only touch the CPU and memory, with no outside contact, and D can do this with the built in tests. He introduced us to test-driven development, emphasizing how it is important how tests fail early, but said it doesn't cover everything and more is needed beyond unit tests - higher level tests are needed, and TDD starts to become BDD - behavior driven development, discussing pros and cons of it, including that it is complicated, but tends to lead to less crufty and better tested code.

He showed how these tests are written with cucumber, a Ruby framework for behavior driven development, and showed how with the Aruba plugin, we can test D without writing ruby code. His slides have example code.

Atila showed his project, Unencumbered, to allow us to write cucumber test steps in D, using user-defined attributes (UDAs) and compile-time reflection to link them together with code. His slides and the talk showed how it is implemented. His future steps include possibly using lambdas rather than named functions, and maybe getting a D only impl. The code is on dub and github.

Dicebok asked: why use cucumber instead of doing it yourself? Atila answered tat he likes the Cucumber model and the reporting tools of offers.

$(P Andrei asked about documentation, to which Atila answered there isn't much for the D parts, but since most of it is the same Cucumber used for Ruby and other languages, you can learn that and be in a good position. $(P John asked about connecting with built-in unitests, to which Atila said since they don't take parameters, they won't work directly, but you can use regular assert in cucumber tests too, so there's no need for an extra test framework on top.

Finally, Erich Gubler, a UVU student, went to talk about his experience using D for his school projects. His slides are here. Editor's note: the live presentation of his things kinda made me dizzy, all kinds of zooming in and out! But his delivery was energetic and positive, very comfortable. He's a better missionary than I am! :P

To finish his undergrad degree, Erich had to write a compiler and virtual machine. He had choice of language and used D to do it after being introduced to it by Chuck Allison, and was left with two overarching feelings: a lot of work goes into the low-level infrastructure we can take for granted, and that D rox! His talk was generally showing just where D rox, with some comparisons to C++.

One of the first things he praised was to!string(some_enum), which converts back and forth from the identifier names in an enum definition:

  enum Foo { bar, baz }
  assert(to!string(Foo.bar) == "bar");
  assert(to!Foo("baz") == Foo.baz);

He showed how short code can get using this. Editor's note: I also used this capability in web code to make convenient whitelists. enum Options { all, valid, options } then use to!Options(user_string) to filter it down to just them - it'd throw an exception if an invalid value was sent automatically. I thought this was so convenient and pretty cool.

He also liked how easy to is to concatenate strings to do more conversions, praising how D removed the obstacles.

He gave a tip: when doing a web search, use the string "dlang" instead of just "d" to get better results.

Continuing his praise for D and its library, he said Phobos raises the bar compared to C++, and just browsing the documentation was a treat. He said he loves std.csv and that std.getopt was so much nicer to use than GNU getopt that he even ported it back to C++ because he missed it so much outside of D.

He said these things are nice because you can get stuff done fast and still optimize later, all in D, and quoted Walter about a fully-functional machine shop on a ship. Erich and Walter said that the time to master comprehensive tools is worth the investment, and D is a fully-loaded shop of comprehensive tools.

Erich then praised rdmd for its ease of use, specifically calling out rdmd -I.. -unittest -main vm.d as an easy way to unit test just one file, without worrying so much about building the whole project. Walter chimed in adding that you should use -cov too.

Erich pointed out a cool tip: you can trace method calls in a single place: the invariant() {} block, which is automatically called before and after each method call of an object (in non-release builds, it is part of the contract programming support in D). Editor's note: using a stack trace print function, calling defaultTraceHandler like I described near the end of my book, you could print not just calls, but whole stack traces with a single point of modification with the invariant block!

Erich said we shouldn't be afraid to break things in the name of progress, provided an easy transition path, and that D needs to be easy to start with, which is mostly a matter of marketing and tutorials at this point. Editor's note: this is a recurring theme, if you do something new in D, please document it, just write what you did, and send it to me. I'll put any tutorials in This Week in D and will organize them too for easy finding, we just need to write them. They don't even need to be great, just the steps you did can help someone follow in those footsteps later.

His conclusion was that D gets obstacles out of the way and is fully-equipped machine shop... in short, it rox for everything.

That concluded the 2015 D Conference. We loitered for a few more hours talking about our individual work, but there was nothing terribly relevant to write up beyond the big announcement: DConf 2016 will be hosted by Sociomantic in Berlin, Germany next year, and they intend to live stream it properly from the beginning rather than the improvised solution we had this year.

I hope you enjoyed these write-ups and encourage you to look through the slides for each talk if you haven't yet, and to watch the videos as they come out. Walter's is already up and the UVU channel is getting more as they are finished. I'll mention them here too when they are up. I will also eventually write a full transcript of my talk, like I did last year and plan to have more from other speakers too, so stay tuned!

Open D Jobs

A page has been added to the D Wiki listing open D jobs. Take a look if you're interested, and add yours if you know of one that is available!

In the community

Community announcements

See more at digitalmars.D.announce.

Significant Forum Discussions

See more at forum.dlang.org and keep up with community blogs at Planet D.

Learn more about D

To learn more about D and what's happening in D: