Language and Performance: Java and D

Table of Contents

• Introduction
• Method
  • Performance Test Problem
  • Measuring Performance
  • Environments
• Results
  • Binary Size
  • Compilation Time
  • Performance Test
  • Runtime Memory Usage
• Conclusions

Introduction

In the previous article, Perspectives on Software Performance, it was discussed how the computational performance and efficient use of RAM can be used to greatly lower the costs associated with solving a given business problem. In this article, the impact of the choice of technology will be further elaborated.

This analysis will focus on a simple representative problem and two programming languages with considerable overlap in the kinds of problems they can solve:

• Java :: Write once, run anywhere.
  • Code compiles into virtual machine instructions, are portable between systems.
  • Java machine instructions are executed by a machine-specific Java Virtual Machine.
  • Heavily object-oriented.
  • Strong tool support.
  • Large community.
• D :: Write fast, read fast, and run fast.
  • Code compiles into machine-specific assembly instructions.
  • Multi-paradigm: iterative, functional, object-oriented, etc.
  • Limited tool support.
  • Small community.

The selection of a tool is very much a pragmatic decision that is specific to the business problem and the company that is solving that problem.

There are many circumstances where one should choose a more popular language:

• A large number of engineers need to be hired. In these cases, the costs of training are reduced when a large number of engineers can be brought in which already have training.
• The problem being solved has little competition, thus, performance is not critical.
• Access to funds is not a problem, thus, cost of operation plays a less significant role than quickly acquiring market share.

Conversely, there are also circumstances where one should choose a more specialized language:

• The problem being solved requires specialist knowledge, and the community of engineers who work on that problem have a favored language.
• The problem being solved is in a competitive environment, thus, performance, speed, and productivity is needed that exceeds the already established competitors.
• Access to funds is restricted, thus, profitability and the cost of operation plays a significant role.

In a nutshell, the best time to choose a less popular, but more specialized, language or tool is when one is trying to beat the competition rather than follow in their footsteps. That being said, there are many specialized tools, and a lack of popularity should be seen as a disadvantage unless there are tangible benefits that make up for it.

For example, if one is building houses and all other construction companies are using hand screwdrivers, going with a power-drill can offer a competitive advantage, despite it not being the popular choice. Some specialized technologies eventually become popular, such as power-drills, while others remain relegated to specialized domains. However, knowledge of technologies, as well as their advantages and disadvantages, is needed in order to make informed decisions when starting to tackle a business challenge.

Method

In order to compare and assess various tools and frameworks, it is important to have a representative problem that is both representative of business problems that may be encountered and that is also free from confounding factors.

Performance Test Problem

In this experiment, the following representative problem was chosen:

This problem contains very little business logic, but enough so that the compiler cannot simply optimize away huge swaths of logic by returning a fixed response.

In Java, a typical setup using the Spring Boot framework can be found here. The main controller has a very simple implementation as shown below:

package com.example.web;

import org.springframework.web.bind.annotation.RestController;
import org.springframework.web.server.ResponseStatusException;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestBody;
import org.springframework.http.HttpStatus;

class UserRequest {
  String username;
}

@RestController
@RequestMapping(path = "/users")
public class WebController {
  @PostMapping("{username}")
  String postUser(@PathVariable String username, @RequestBody UserRequest userRequest) {
    if (username != userRequest.username) {
      throw new ResponseStatusException(HttpStatus.BAD_REQUEST, "Username does not match!");
    }
    return "OK";
  }
}
Code language: Java (java)

In D, a typical setup using the Vibe.d framework can be found here. The main controller has an implementation very similar to the one in Java.

module appcontroller;

import vibe.web.common : path;
import vibe.http.common : HTTPStatusException;
import vibe.http.server : HTTPServerRequest;
import vibe.data.json : deserializeJson;

struct UserRequest {
  string username;
}

@path("/users")
class AppController {
  @path(":username")
  string postData(string _username, HTTPServerRequest request) {
    UserRequest userRequest = request.json().deserializeJson!UserRequest();
    if (_username != userRequest.username) {
      throw new HTTPStatusException(400, "Username does not match!");
    }
    return "OK";
  }
}
Code language: D (d)

Measuring Performance

There are various factors that play into performance, including executable binary size, memory consumption, and most importantly, how quickly requests can be processed.

In order to measure the performance of processing requests, the tool Siege will be used. Siege is a highly efficient and performant load testing tool. The following test parameters will be used:

• Number of concurrent users: 25
• Number of sessions per user: 1000
• Number of requests per session: 3
  • HTTP POST http://localhost:8080/users/jim
    • 200 OK, request body matches path
  • HTTP POST http://localhost:8080/users/bob
    • 200 OK, request body matches path
  • HTTP POST http://localhost:8080/users/james
    • 400 BAD_REQUEST, request body does not match path

In order to focus primarily on the performance of the servers themselves and not on network speeds, Siege will be run from within the same computer as the web servers, accessing them via localhost.

Environments

The environment used to execute tests is specified here in order to make it easier to replicate tests as well as to update them with additional data in the future.

• Computer
  • CPU:
    • Type: Intel(R) Core(TM) i7-3630QM CPU
    • Speed: 2.40GHz
    • Total Cores: 4
    • Total Threads: 8
    • Cache Size per Thread: 6144 KB
  • Memory:
    • Size: 16 GiB
    • Speed: 1600 MT/s
• Operating System
  • Ubuntu 20.04.4 LTS
• Compilers
  • Java Compilers:
    • OpenJDK v11.0.10
  • D Compilers:
    • DMD v2.098.1 :: The reference D compiler from Digital Mars.
    • LDC v1.20.1 :: The LLVM D compiler.

The D programming language has multiple compilers, each of which offer various advantages:

• DMD :: Fast compilation times, great during development.
• GDC :: A GCC-based compiler with excellent integration with GDB.
• LDC :: An LLVM-based compiler with strong optimization.

Results

Binary Size

The size of a binary loosely relates to the amount of data being downloaded during each build and deployment of a service. The more data that needs to be transmitted, the longer it takes to complete a build.

The size of the binary was determined using the Linux command: ls -l

Language (Build)	Binary Size
Java	16.8 MiB *
DMD (debug)	36.7 MiB
DMD (release)	12.7 MiB
LDC (debug)	18.4 MiB
LDC (release)	3.8 MiB

* A fully operational Java program also requires a JRE which needs an additional 125 MiB.

Using only the size of the Java binary alone, without the JRE, the largest D binary from DMD in debug mode requires 118% more disk space than Java. The smallest D binary from LDC in release mode requires 23% as much disk space as Java.

Taking into account the size of the JRE, the largest D binary from DMD in debug mode requires approximately 26% as much disk space as Java. The smallest D binary from LDC in release mode requires approximately 2.7% as much disk space as Java.

Compilation Time

The compilation times were determined using the Linux time command. To determine the “full” compilation time, the projects were cleaned and compiled without any pre-compiled artifacts. The download time of dependencies was excluded from these results. This time represents the typical compilation time for building artifacts for deployment.

Java projects were compiled with the command: mvn package

D projects were compiled with the command: dub build -b debug or dub build -b release

To determine the “incremental” compilation time, a single file was modified in each project and all cached data was preserved. This time represents the typical recompilation times during development.

Language (Build)	Compilation Time (full)	Compilation Time (incremental)
Java	real 0m9.816s user 0m34.764s sys 0m0.994s	real 0m9.421s user 0m34.838s sys 0m1.004s
DMD (debug)	real 0m28.576s user 0m23.415s sys 0m4.654s	real 0m7.451s user 0m6.284s sys 0m1.169s
DMD (release)	real 1m7.996s user 1m3.314s sys 0m4.417s	real 0m6.889s user 0m5.941s sys 0m0.952s
LDC (debug)	real 0m56.229s user 0m51.116s sys 0m4.788s	real 0m5.290s user 0m4.453s sys 0m0.793s
LDC (release)	real 1m41.333s user 1m37.546s sys 0m3.547s	real 0m4.918s user 0m4.302s sys 0m0.575s

Java is actually especially interesting in this test because the full and incremental compilation times are very similar. Additionally, the Java compiler makes heavy use of multi-threading, with almost 4x as much user time used as real time (indicating 4 processes acting in parallel).

For incremental development, all compilers and all build types for D offer reductions in compilation times ranging from 21% to 48%. These are small amounts of time, from 2 to 4 seconds, however, it can make for a more responsive and productive development environment depending on the engineer.

The various D compilers vary in speed considerably and are quite fast, but compilation times could be further improved through the use of multi-threading. Also, build tool dub compiles all the dependencies from source, which greatly increases the full compilation time.

Another interesting observation about D compilation times is that the strong reduction in the size of binaries associated with release builds actually results in compilation times of incremental builds that are even lower than those of debug builds.

Performance Test

Each server was tested in its performance handling request as described in Measuring Performance. To recap, the test simulates 25 concurrent users, who make 1000 sessions, and each session consists of 3 HTTP POST requests. The time it takes the server to process these 75,000 requests is measured.

Language (Build)	Transaction Rate
Java	5165.29 trans/sec
DMD (debug)	157.40 trans/sec
DMD (release)	9363.30 trans/sec
LDC (debug)	180.19 trans/sec
LDC (release)	10683.76 trans/sec

The rate of processing transactions is significantly lower for debug builds in D, using both LDC and DMD, compared to Java. However, the rate of processing transactions is significantly higher using release builds using DMD (+81%) and LDC (+107%).

Runtime Memory Usage

The amount of memory that a running server takes is also an important consideration. Between CPU, RAM, and network usage, the number of physical computers needed to implement a cluster is determined.

The amount of RAM needed was determined by using the Linux command: ps -o pid,tty,stat,time,%mem,rss,trs,vsz,command <pid>

Language (Build)	Resident / Virtual Memory Usage
Java	278.564 MiB / 10,002.048 MiB
DMD (debug)	12.768 MiB / 56.876 MiB
DMD (release)	11.972 MiB / 55.828 MiB
LDC (debug)	12.112 MiB / 514.912 MiB
LDC (release)	11.396 MiB / 513.780 MiB

Compared to Java, the D server compiled with DMD uses 96% less resident memory and 99% less virtual memory. The D server compiled with LLVM uses 96% less resident memory and 95% less virtual memory.

Conclusions

The choice of programming languages can make significant changes in performance along many dimensions ranging from a near doubling of runtime performance, using only 5% as much RAM, having faster compilation times, and smaller binaries that can be more quickly released.

However, the use of special purpose tools and languages comes with the very real challenges of finding engineers who are either already familiar with them or are capable of learning from them. Furthermore, the smaller community size means that finding support from the online community is more difficult, and the availability of answers to questions one encounters during development is lower.

Languages like D benefit from having an extremely similar syntax to languages like Java, and thus has a very quick learning curve. Companies that have smaller numbers of experienced engineers who require less assistance on common tasks can find significant competitive advantages by using specialized languages like D.