Java Microbenchmark Harness is part of the Java 9 release.

JMH takes care of JVM warm-up and code-optimization paths, making benchmarking as simple as possible and excluding as many test influences as possible.

It gives you an easy way of specifying how many warmup and test iterations you want to do.

It will also fork your test so that code-optimization for one test does not influence the results of an other test.

And it allows you to specify how much memory you want to give to the JVM such that all tests have the same resources.

 

Example

Here is a code example that compares different ways of looping through a set of data. Will the Java8 way of working also give me a performance benefit?

Let’s test it!

@BenchmarkMode(Mode.AverageTime)
@OutputTimeUnit(TimeUnit.MILLISECONDS)
@State(Scope.Benchmark)
public class BenchmarkLoops {

    @Param({"5000000"})
    private int sizeOfTestData;

    private List<String> testData;

    public static void main(String[] args) throws RunnerException {

        Options opt = new OptionsBuilder()
                .include(BenchmarkLoop.class.getSimpleName())
                .forks(1)
                .jvmArgs("-Xms2G", "-Xmx2G")
                .warmupIterations(3)
                .measurementIterations(3)
                .build();

        new Runner(opt).run();
    }

    @Setup
    public void setup() {
        testData = createTestData();
    }

    // Blackhole has convenient methods to consume data
    @Benchmark
    public void loopFor(Blackhole bh) {
        for (int i = 0; i < testData.size(); i++) {
            String s = testData.get(i);
            bh.consume(s);
        }
    }

    @Benchmark
    public void loopWhile(Blackhole bh) {
        int i = 0;
        while (i < testData.size()) {
            String s = testData.get(i);
            bh.consume(s);
            i++;
        }
    }

    @Benchmark
    public void loopForEach(Blackhole bh) {
        for (String s : testData) {
            bh.consume(s);
        }
    }

    @Benchmark
    public void loopIterator(Blackhole bh) {
        Iterator<String> iterator = testData.iterator();
        while (iterator.hasNext()) {
            String s = iterator.next();
            bh.consume(s);
        }
    }

    @Benchmark
    public void loopIntStream(Blackhole bh) {
        IntStream.range(0, testData.size()).forEach(i -> bh.consume(testData.get(i)));
    }

    @Benchmark
    public void loopStream(Blackhole bh) {
        testData.forEach(bh::consume);
    }

    private List<String> createTestData() {
        List<String> data = new ArrayList<>();
        new Random().ints().limit(sizeOfTestData).forEach(i -> data.add("Number: " + i));
        return data;
    }

}

At my machine the result is the following (the lower the Score the better the result):

Benchmark                    (sizeOfTestData)  Mode  Cnt   Score    Error  Units
BenchmarkLoop.loopFor                 5000000  avgt    3  45.443 ± 19.190  ms/op
BenchmarkLoop.loopForEach             5000000  avgt    3  47.843 ± 42.055  ms/op
BenchmarkLoop.loopIntStream           5000000  avgt    3  27.522 ±  1.673  ms/op
BenchmarkLoop.loopIterator            5000000  avgt    3  50.847 ± 61.170  ms/op
BenchmarkLoop.loopStream              5000000  avgt    3  50.568 ± 26.601  ms/op
BenchmarkLoop.loopWhile               5000000  avgt    3  47.252 ± 29.573  ms/op

I have to say that I did not expect this result. We see that the loopIntStream() method was by far the most performant.

While the java8 Streams API often provides more visibility, it mostly comes with a performance benefit. At least, to my knowledge.

This got me thinking.. Maybe I get this result because in that method I do not store the result of testDate.get(i) in a local variable.

So I changed the method to:

@Benchmark
public void loopIntStream(Blackhole bh) {
    IntStream.range(0, testData.size()).forEach(i -> {
        String s = testData.get(i);
        bh.consume(s);
    });
}

And ran the tests again. This gave me:

Benchmark                     (sizeOfTestData)  Mode  Cnt   Score     Error  Units
BenchmarkLoops.loopFor                 5000000  avgt    3  38.824 ±  18.180  ms/op
BenchmarkLoops.loopForEach             5000000  avgt    3  39.720 ±   6.830  ms/op
BenchmarkLoops.loopIntStream           5000000  avgt    3  49.023 ± 118.128  ms/op
BenchmarkLoops.loopIterator            5000000  avgt    3  48.342 ±  50.839  ms/op
BenchmarkLoops.loopStream              5000000  avgt    3  51.381 ±  22.316  ms/op
BenchmarkLoops.loopWhile               5000000  avgt    3  47.829 ±  90.582  ms/op

That’s a more logical result. Nonetheless it shows how small things can drastically influence your test results! So pay attention to what you are measuring!