This Week in D July 12, 2015

Welcome to This Week in D! Each week, we'll summarize what's been going on in the D community and write brief advice columns to help you get the most out of the D Programming Language.

The D Programming Language is a general purpose programming language that offers modern convenience, modeling power, and native efficiency with a familiar C-style syntax.

This Week in D has an RSS feed.

This Week in D is edited by Adam D. Ruppe. Contact me with any questions, comments, or contributions.

Statistics

DConf 2015

The videos are now all released. They can be found on UVU's youtube channel and on the DConf website talk pages.

Open D Jobs

A page has been added to the D Wiki listing open D jobs. Take a look if you're interested, and add yours if you know of one that is available!

In the community

Community announcements

See more at digitalmars.D.announce.

Significant Forum Discussions

Wait, what? What is AliasSeq? was a long bikeshedding discussion - people arguing over the name of a trivial, but useful line of code: alias whatever_name(T...) = T;. People argued it isn't a tuple because it automatically expands, it isn't a list because it is indexable, it isn't arguments because it might be parameters, and similar for post after post after post.

No decision seems to have been made. The old name for that in Phobos was TypeTuple which nobody really loved. Essentially, it is just a list of random compile time things.

Community Interview

Our interview this week is Etienne Cimon, again courtesy of Joakim.

Etienne Cimon is an active contributor on dub, the D repository for third-party libraries. He has ported C++ libraries to D, like the Botan crypto library and libhttp2 from nghttp2, and written new D libraries, such as the libasync event loop library and memutils, which provides some allocators and containers that employ them.

Q: How was the experience writing your own D libraries? What stood out for you when writing in D, as opposed to other languages you used before?

A: I wanted speed, and I got speed, and it was easy and short. At a certain point, it essentially gives you C speed with javascript syntax. What stood out most is the delegate type, it can behave in many different ways, it even lets you write a callback somewhat like in javascript, e.g.

auto func2 = (string str) { return str ~ " you!"; }; // the ~ is the only allocation in this code! 
auto func = { return func2("hey"); }; 
writeln(func());  // prints: "hey you!";

I wanted to put that ease into an async library and it took me less than a month. I interfaced it with Windows/Linux/BSD/Mac easily because there were bindings to all the platforms in druntime and there's lots of help around to achieve it. I felt like a kid having Superman by my side in school.

I won't go into the details because you couldn't cover all of the language features in four years of full-time schooling. But it's easy to stop at a subset of them if you don't need it all.

Q: Can you go into the details a bit more? Obviously you weren't using all of D's features, but what stood out for you as most helpful or useful compared to other languages, other than delegates?

A:
- static if, static assert, version, traits and mixins. When I discovered those, I was going nuts with C++ metaprogramming, which relies entirely on type inference, recursion, and <, >, nesting to do anything at all. I studied boost::phoenix, boost::spirit, boost::serialization, and I believe the implementation details of those libraries would be much simpler in D. In fact, it would look like very regular OOP. Metaprogramming in D is a little like reflection in other languages, minus the additional CPU cycles: the compiler spits out the most optimal instructions in every case. That's the most attractive part of course: not having to write the optimal boilerplate for every scenario. I used this to develop the alloc!MyType(args) and free(my_type) functions, which mimic the functionality of new in the compiler thanks to the introspective ability of the compiler.

- UFCS and the package protection keyword. I can extend classes and structs using static function declarations in different modules, e.g. definition: `void echo(HTTPServerResponse res, Object o) { res.bodyWriter.write(o.toString()); }`
usage: res.echo(obj);. No need for inheritance, and it also becomes very useful for generic methods with the right template overloading. The .to!T method is the best example, excellent for chaining calls.

Q: How was the experience porting C++ libraries like botan or nghttp2? Talk about the process and what kind of changes you had to make, including what the tougher or more tedious parts were.

A: On the plus side, these libraries didn't rely on C++ or C external libraries which made it a LOT easier than it should have been. However, I did come up with a few problems.

The hard:
- The naming convention takes very long to change. Yes, going snake_case to camelCase is important for readability. :)

- The C++ STL didn't have a corresponding library in D, so I wrote memutils. The Vector/Map/Reference counting/Unique/RBTree with custom allocators had to be written with similar behavior, and with a GC monitoring everything I needed to perfectly understand the side effects.. thankfully it's a simple garbage collector!

- The macros are very difficult to reason about, especially things like this, which will be templated and call in mixins like this.

The easy:
- No difficulty at all with porting a C library.

- Sounds crazy, but the whole process of building the library was very much simplified and I managed to flush the build scripts from C++ and make a simple configuration file for dub instead.

- The whole thing is very repetitive. Writing a lot of regex to search & replace the types or function names and moving definitions under the corresponding header file declarations was the longest part.

- I loved the refactoring of the libraries. For C, you could take away most of the names by enclosing into structs, which makes for cleaner function bodies e.g. going from session_inbound_frame_reset(nghttp2_session *session) to reset().

Q: Please tell us about your background: who you are and where you're from, what programming languages you used before D, and take us from your experience first discovering and using D to porting and writing your own D libraries.

A: I started programming when I was nine years old in Ontario (Burlington), I got a 486 that had 33mhz regular, 66mhz on turbo. Made me think it was extremely powerful, like those cars with turbo engines. But my neighbor brought in a Pentium 133 mhz and it made me feel totally envious. However, the guys there were 4-6 years older than me and some of them were experts at it. I quickly learned to go on IRC, most notably #warez97 or stuff like that, where I discovered a lot of software by running an ftp server called serv-u with 1:2 upload ratio and advertising it on IRC. I had the first cable modem in the block and it was like 18-60KB/s instead of the 3KB/s I'd get on a dialup. Awesome times.

So I kept on going on the computer 8 hours per day after school and even at school (I was in protic) until I knew everything I could. HTML, javascript, perl, visual basic, php, C# later.. I worked for the University of Alberta at 17 years old on a web software project too (at NANUC, very nice bunch). I could say things went pretty well for me: I went through engineering school while running my software company doing e-commerce software or custom stuff mostly in the Montreal area (Quebec). Most of the stuff I made was by myself and usually along the lines of a partnership with a designer at definiteimage.com. My most important developments have followed the companies through their expansion and still run to this day.

I started building my own line of private projects 4 years ago and run on my own income streams by now.

I was mad at PHP/MySQL because it was running so slow, so I decided to go lower level into C and C++. I read a lot on the processor intrinsics and most importantly the article from Ulrich Drepper, What everyone should know about memory. I decided to port some projects to C++ but became overwhelmed at the complete disorganization of the frameworks in that community, how heavy (megabytes of DLL just for utf-8?) each library is.

I found D when searching for HTTP libraries, I found that curl had a sane interface on it. It was lightweight, complete, had a good license, sane errors: I was impressed. Then I remember glancing over the vibed.org webpage and seeing an entire high-level HTTP framework in a low-level language! I had a hard time imagining it was even possible, when boost wanted me to remember my context switches through strides, or I don't remember what they called them. I decided never to look back, and S¶nke quickly became a great inspiration to me in learning how to program high-level code in a low-level language. However, I think there was still too much C and C++ in there, which is why I decided to start writing low-level libraries anyways.

Q: Do you make money writing all this D code? What do you do for a living?

A: I develop private software, mostly in the social networking area. I can't speak of it sadly. The web stack in D that I wrote is going to be at the center of my career and developments for decades however. I also write a lot of javascript, SQL, java, C and shell scripts.

Q: You've said on the forums that, "Nothing beats D in my opinion. It's 20 years ahead of everything out there." Please expand on that: why do you think it's so great?

A: My conclusion by elimination and from a business perspective is that D is the most appropriate "glue" language (the primary application language) for small to large projects where you care about speed.

My criteria for a good "glue" language is that it has to be C-style so that your whole team can pick it up quickly; it has to be fast; have a lot of features, including static types; compile to machine code; it has to do what you want without becoming dirty; it needs to have at least a good compiler that you can fix yourself; run on all platforms or potentially so, flexible enough to get it in the direction you need.

Why "glue" language? Shell scripts are usually the solution for 99% of server administration tasks, and SQL is the solution for 99% of data analysis, jade templates for HTML, R is the solution for statistics, some people like to write a quick script in python or javascript to play around with X and Y. Some want their database in sqlite written with C. Why the f*** not? Glue it to your main application and see it shine even more, ie. put that shit in a D module, call it a day. And if it doesn't shine for what you need, go make a pull request on github and a team of world-class experts is going to give you a bunch of constructive comments until it does.

If the world isn't ready for D to take over as a main language, it'll probably be in 20 years when Apple releases the iLang or whatever and all the vegan fanboys go crazy about how everything shines in the iLang.

Q: What problems have you had with D? What features do you dislike?

A: I really want both a shared and thread-local GC. We need to do something about that shared attribute, new shared T => shared GC, new T => thread-local GC. I hope I get more support for that. Non-locking solutions are the answer to perfect concurrency.

There were a few things lacking, like the COFF symbol limits in DMD. You usually just fix those things, but there's always those other things that were hindering development and I didn't fix but worked around them because I lacked the time to develop a patch.

- I think the lack of backtrace is what got to me the most. I had to resort to addr2line most of the time because DMD spits out the instruction addresses on the call stack. I tried some tools that did this automatically but it didn't always work. GDB also crashed a few times becase of the D symbols. I ended up relying on windbg (x64) and it became my tool of choice. It feels incredibly empowering to use it in development: I'll probably get burned for saying this but a modern GUI with a good resolution is much more convenient than a small console window.

- It's practically impossible to find leaks in the GC. Valgrind will look at malloc/free which will show the first time the pointer was allocated. I had to implement my own software-level leak detector via mapping allocations in memutils' DebugAllocator. The idea is to print all allocation pointers and sizes in a file, and show the remaining pointer addresses at the end of the program along with their size, then search through the allocations for each of them, and reiterate this process but with a stack trace every time the matching pointer sizes are allocated (I haven't had time to look into the new GC profiling features in git master).

Q: When I first emailed you, you mentioned that you just got vibe.d to "run stable" with your new D replacement libraries. Can you go into what that means, ie how you're testing that it's stable?

A: I ran 2 vibe.d servers using these settings:
Firefox <= http/2 => Vibe.d static file server
Firefox <= HTTP/1.1 => Vibe.d Reverse Proxy <= http/2 & botan TLS => Vibe.d static file server

So basically, the reverse proxy is an HTTP/2 secure client established through either upgrade, ALPN or prior knowledge. The certificates and cipher suites were alternated to different sets. A regular website with varying payload sizes and possible errors was served. Javascripts could be 150kb, image files up to 1MB, and form uploads were also tested.

I was using the firefox window debugger to analyse the response times and return codes.

This allowed me to debug many underlying components in what we can call "Integration testing". For example, the requests would often leak tasks for no reason: I found that a signal was being unregistered because a stack algorithm was duplicating an event every time the stack grew (double policy).

I also tested closing the connection at every possible moment during protocol negotiations. The very complex and unprecedented scientific methodology used to achieve this was to hold F5 in Firefox. I found that some errors were being generated inside constructors and causing leaks, for example.

Q: You also said you implemented "release-mode debugging symbols for web server call stack and a request capture for debugging:" can you go over what you did there?

A: The release mode symbols was achieved by appending the result of the compiler call __PRETTY__FUNCTION__ to a global fiber-specific call stack in a red-black tree, and unwinding the stack when the scope succeeded. When it failed, a counter was incremented to register this call stack in exceptions, ie. on the error page in the HTTP server.

This was simplified by adding mixin(Trace); at the beginning of tracked functions.

The capture was achieved by allowing each task to hold "Breadcrumbs", e.g. 127.0.0.1:4459 > GET > /url/. It also allowed tasks to hold a "Name" e.g. HTTP Server Request. You could then use a "Keyword" to assign data to be captured.

This was simplified by adding a `mixin(OnCapture!("HTTP Request Headers", "req.headers.toString()"));`.

The capture interface can be seen here.

This allows me to see not only the request and response headers, but also the request and response form data, the pretty request and response json data, the certificate information ie. algorithm, public key, issuer, etc.

This is very useful for debugging purposes when also available in the HTTP Client. I can therefore write a library to interface with the instagram public API and see what Json was returned. It is also available on a live site, so if my vibe.d application stops working correctly for a certain query, I can just launch the capture interface and finally see what the f*** is going on in this shit. I have a family ffs, I don't want to lose my entire existence debugging an invalid json payload.

Q: Can you talk about the performance of your new vibe.d stack with D libraries: how fast is it? Particularly in relation to the original C++ libraries, if you know.

A: No idea, I'm not going to benchmark it on DMD because it's probably 2-3x slower. It does compile about 10x faster, <10s vs >1 min. I think LDC will be a little slower to compile but a much better match, no idea how it'll do but it could definitely be faster than the C/C++ implementations. I will make the entire stack compile with LDC once druntime/phobos 2.067+ is available in LDC, because some things couldn't work on previous versions.

Q: What are your plans for future D libraries or features, whether C++ ports or written in D?

A: I intend to write quite a few other ports. One of them is a DNS server ported from Go. It would however contain a load distribution round-robin policy and some specific DDoS countermeasures I thought about. DDoS attackers usually don't cache the DNS records because on any decent service, new servers are dynamically advertised as others are shut down. So the countermeasure would be to redirect the detected attacker to another attacker's IP through the A records. Heheh

Another library would be for a Quic implementation. Once (if) it becomes standard, would be nice to have due to the huge number of roundtrips in TCP + TLS + HTTP/2 connection negotiation.

Of course, I'll keep the rest of the stack updated.

Learn more about D

To learn more about D and what's happening in D: