<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom" xmlns:media="http://search.yahoo.com/mrss/"><generator uri="https://jekyllrb.com/" version="4.4.1">Jekyll</generator><link href="https://mehmandarov.com/feed.xml" rel="self" type="application/atom+xml"/><link href="https://mehmandarov.com/" rel="alternate" type="text/html"/><updated>2026-06-21T11:30:11+02:00</updated><id>https://mehmandarov.com/feed.xml</id><title type="html">Rustam Mehmandarov</title><subtitle>A blog about technology and related stuff</subtitle><author><name>Rustam Mehmandarov</name></author><entry><title type="html">Unknown JSON fields in Java REST clients: JSON-B, Jackson, Quarkus, and Spring Boot</title><link href="https://mehmandarov.com/unknown-json-fields-in-java-rest-clients/" rel="alternate" type="text/html" title="Unknown JSON fields in Java REST clients: JSON-B, Jackson, Quarkus, and Spring Boot"/><published>2026-06-21T09:00:00+02:00</published><updated>2026-06-21T09:00:00+02:00</updated><id>https://mehmandarov.com/unknown-json-fields-in-java-rest-clients</id><content type="html" xml:base="https://mehmandarov.com/unknown-json-fields-in-java-rest-clients/"><![CDATA[<p><em>You call an API with the MicroProfile REST Client, map the response onto a small DTO, and one day the API starts returning a few extra fields you never asked for. Does your client shrug and carry on, or does it blow up with a deserialization error? The honest answer is &#8220;it depends on your JSON provider&#8221; &#8211; and the defaults are not the same across the board. Let&#8217;s pin down what actually happens, and point to the spec or docs for each case.</em></p>

<ul>
  <li><a href="#introduction">Introduction</a></li>
  <li><a href="#why-this-bites-people">Why this bites people</a></li>
  <li><a href="#show-me-the-code">Show me the code</a></li>
  <li><a href="#1-the-default-json-b-yasson">1. The default: JSON-B (Yasson)</a></li>
  <li><a href="#2-jackson-strict-by-default">2. Jackson: strict by default</a></li>
  <li><a href="#3-quarkus-jackson-but-lenient">3. Quarkus: Jackson, but lenient</a></li>
  <li><a href="#4-spring-boot-also-lenient">4. Spring Boot: also lenient</a></li>
  <li><a href="#a-note-on-the-other-direction-server-receiving-extra-fields">A note on the other direction (server receiving extra fields)</a></li>
  <li><a href="#summary-comparison">Summary Comparison</a></li>
  <li><a href="#conclusion">Conclusion</a></li>
  <li><a href="#whats-next">What&#8217;s Next?</a></li>
</ul>

<hr />

<h2 id="introduction">Introduction</h2>

<p>Imagine a small REST client. You are consuming a &#8220;room&#8221; resource from some conference API, and you only care about three fields &#8211; <code class="language-plaintext highlighter-rouge">id</code>, <code class="language-plaintext highlighter-rouge">name</code>, and <code class="language-plaintext highlighter-rouge">capacity</code>:</p>

<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">public</span> <span class="kd">record</span> <span class="nf">Room</span><span class="o">(</span><span class="nc">String</span> <span class="n">id</span><span class="o">,</span> <span class="nc">String</span> <span class="n">name</span><span class="o">,</span> <span class="kt">int</span> <span class="n">capacity</span><span class="o">)</span> <span class="o">{</span> <span class="o">}</span>
</code></pre></div></div>

<p>You wire it up with the <a href="https://download.eclipse.org/microprofile/microprofile-rest-client-3.0/microprofile-rest-client-spec-3.0.html">MicroProfile REST Client</a>:</p>

<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nd">@RegisterRestClient</span><span class="o">(</span><span class="n">baseUri</span> <span class="o">=</span> <span class="s">"https://conf.example.com/api"</span><span class="o">)</span>
<span class="kd">public</span> <span class="kd">interface</span> <span class="nc">RoomsClient</span> <span class="o">{</span>

    <span class="nd">@GET</span>
    <span class="nd">@Path</span><span class="o">(</span><span class="s">"/rooms/{id}"</span><span class="o">)</span>
    <span class="nd">@Produces</span><span class="o">(</span><span class="nc">MediaType</span><span class="o">.</span><span class="na">APPLICATION_JSON</span><span class="o">)</span>
    <span class="nc">Room</span> <span class="nf">getById</span><span class="o">(</span><span class="nd">@PathParam</span><span class="o">(</span><span class="s">"id"</span><span class="o">)</span> <span class="nc">String</span> <span class="n">id</span><span class="o">);</span>
<span class="o">}</span>
</code></pre></div></div>

<p>This works fine. Then, a few sprints later, the API team <strong>adds</strong> <code class="language-plaintext highlighter-rouge">building</code>, <code class="language-plaintext highlighter-rouge">floor</code>, and <code class="language-plaintext highlighter-rouge">accessibility</code> to the room payload. Your DTO still declares three fields. The response now looks like this:</p>

<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w">
  </span><span class="nl">"id"</span><span class="p">:</span><span class="w"> </span><span class="s2">"room-7"</span><span class="p">,</span><span class="w">
  </span><span class="nl">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Hall A"</span><span class="p">,</span><span class="w">
  </span><span class="nl">"capacity"</span><span class="p">:</span><span class="w"> </span><span class="mi">120</span><span class="p">,</span><span class="w">
  </span><span class="nl">"building"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Main"</span><span class="p">,</span><span class="w">
  </span><span class="nl">"floor"</span><span class="p">:</span><span class="w"> </span><span class="mi">2</span><span class="p">,</span><span class="w">
  </span><span class="nl">"accessibility"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"wheelchair"</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="w"> </span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>

<p>The question is simple: what does <code class="language-plaintext highlighter-rouge">getById("room-7")</code> do now? And the answer, annoyingly, is that it depends entirely on which JSON provider is doing the deserialization. The MicroProfile REST Client spec does not decide this for you &#8211; it delegates the actual JSON binding to whatever provider is on the classpath.</p>

<h2 id="why-this-bites-people">Why this bites people</h2>

<p>This is worth a whole post because the default behaviour is <em>inconsistent</em> between providers, and the failure shows up at the worst possible time &#8211; in production, when someone else&#8217;s API changes underneath you.</p>

<p>A JSON library can reasonably do one of two things when it meets a field that has no home in your DTO:</p>

<ul>
  <li><strong>Be lenient (tolerant reader):</strong> ignore the unknown field and move on. This means an additive change on the server side does not break your client.</li>
  <li><strong>Be strict:</strong> treat an unknown field as a mistake worth reporting, and throw.</li>
</ul>

<p>This idea is not something REST people invented later. It goes back to early Internet protocol design: TCP&#8217;s <a href="https://www.rfc-editor.org/rfc/rfc793#section-2.10">robustness principle</a> (<a href="https://en.wikipedia.org/wiki/Robustness_principle">overview</a>) says to be conservative in what you send and liberal in what you accept from others. For this particular JSON-client case, the practical reading is simple: if the response gives you all the fields you asked for, extra fields should usually be ignored by the consumer. The modern caveat is important, though: this is not a license to accept malformed or unsafe input. Newer <a href="https://www.rfc-editor.org/rfc/rfc9413">protocol guidance</a> explicitly warns that applying the robustness principle too broadly can create interoperability and security problems.</p>

<p>Neither is wrong. But you really want to <em>know</em> which one you have, because the strict default is the one that turns a backwards-compatible server change into a client-side outage.</p>

<blockquote>
  <p>&#128161; <em><strong>Note:</strong> &#8220;Additive response changes should be safe&#8221; is one of the practical compatibility expectations of REST-style JSON APIs. It only holds if your consumers are tolerant readers. A strict deserializer quietly opts you out of that contract.</em></p>
</blockquote>

<h2 id="show-me-the-code">Show me the code</h2>

<p>I have added a small demo to my <a href="https://github.com/mehmandarov/api-guide-java">API Guide for Java</a> repository. The endpoint in <a href="https://github.com/mehmandarov/api-guide-java/blob/main/src/main/java/com/mehmandarov/confapi/unknownfields/UnknownFieldsResource.java"><code class="language-plaintext highlighter-rouge">UnknownFieldsResource.java</code></a> serves a deliberately over-stuffed room payload at <code class="language-plaintext highlighter-rouge">GET /api/unknown-fields/{id}</code> (the six fields from the introduction), and the <a href="https://github.com/mehmandarov/api-guide-java/blob/main/src/test/java/com/mehmandarov/confapi/unit/Ch7_UnknownFieldsTest.java"><code class="language-plaintext highlighter-rouge">Ch7_UnknownFieldsTest</code></a> unit test shows what each provider does when that payload is mapped onto the three-field <code class="language-plaintext highlighter-rouge">Room</code>. Below, I walk through the defaults and the switch that changes each one.</p>

<hr />

<h2 id="1-the-default-json-b-yasson">1. The default: JSON-B (Yasson)</h2>

<h3 id="what-it-looks-like">What it looks like</h3>

<p>On a typical Jakarta EE / MicroProfile stack without Jackson &#8211; think Open Liberty, Helidon, Payara, and friends &#8211; JSON mapping is commonly handled through <strong>JSON-B</strong>. <a href="https://eclipse-ee4j.github.io/yasson/">Yasson</a> is the JSON-B reference implementation, and it is also what the demo test uses.</p>

<p>The good news: JSON-B <strong>ignores unknown properties by default</strong>. The <code class="language-plaintext highlighter-rouge">Room</code> record above deserializes happily, <code class="language-plaintext highlighter-rouge">building</code> and <code class="language-plaintext highlighter-rouge">floor</code> are dropped on the floor, and your client keeps working.</p>

<p>This is not an accident or an implementation detail of Yasson &#8211; it is in the spec. The <a href="https://jakarta.ee/specifications/jsonb/3.0/jakarta-jsonb-spec-3.0.html">Jakarta JSON Binding specification</a> states that during deserialization, any JSON property that does not map to a class member is ignored.</p>

<h3 id="how-to-call-it">How to call it</h3>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>curl <span class="nt">-X</span> GET http://localhost:8080/api/unknown-fields/room-7 <span class="se">\</span>
  <span class="nt">-H</span> <span class="s2">"Accept: application/json"</span>
</code></pre></div></div>

<div class="language-http highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="err">GET http://localhost:8080/api/unknown-fields/room-7
Accept: application/json
</span></code></pre></div></div>

<p>The HTTP endpoint still returns the six-field payload. The important part happens on the client side: JSON-B maps the fields it knows about into <code class="language-plaintext highlighter-rouge">Room</code> and ignores <code class="language-plaintext highlighter-rouge">building</code>, <code class="language-plaintext highlighter-rouge">floor</code>, and <code class="language-plaintext highlighter-rouge">accessibility</code>.</p>

<p><strong>&#9989; Pros:</strong></p>
<ul>
  <li>Tolerant reader by default &#8211; additive server changes don&#8217;t break you.</li>
  <li>No configuration needed; it&#8217;s the platform default.</li>
  <li>Matches the behaviour most people <em>expect</em> from a REST client.</li>
</ul>

<p><strong>&#10060; Cons:</strong></p>
<ul>
  <li>If you <em>want</em> strictness (e.g. to catch a typo in a field name during development), JSON-B gives you less help there.</li>
  <li>Silently dropping fields can hide the fact that the API has grown, and you&#8217;re missing data you might actually want.</li>
</ul>

<p>&#128269; <em><strong>However:</strong> &#8220;Lenient by default&#8221; is the behaviour you usually want for a consumer. Just be aware it is a deliberate choice &#8211; you are trading early failure for compatibility.</em></p>

<hr />

<h2 id="2-jackson-strict-by-default">2. Jackson: strict by default</h2>

<h3 id="what-it-looks-like-1">What it looks like</h3>

<p>The moment you use an unconfigured Jackson mapper or a bare Jackson provider &#8211; for example a plain <code class="language-plaintext highlighter-rouge">ObjectMapper</code>, <code class="language-plaintext highlighter-rouge">resteasy-jackson</code>, or <code class="language-plaintext highlighter-rouge">jersey-media-json-jackson</code> without framework-level configuration &#8211; the default <strong>flips</strong>.</p>

<p>Jackson&#8217;s <code class="language-plaintext highlighter-rouge">ObjectMapper</code> enables <a href="https://fasterxml.github.io/jackson-databind/javadoc/2.18/com/fasterxml/jackson/databind/DeserializationFeature.html#FAIL_ON_UNKNOWN_PROPERTIES"><code class="language-plaintext highlighter-rouge">DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES</code></a> <strong>by default</strong>. An unknown field throws <code class="language-plaintext highlighter-rouge">UnrecognizedPropertyException</code> (a subclass of <code class="language-plaintext highlighter-rouge">JsonMappingException</code>), which typically surfaces through the REST Client as a response-processing/deserialization failure. Your three-field <code class="language-plaintext highlighter-rouge">Room</code> no longer deserializes the six-field payload &#8211; it fails.</p>

<p>There are three common ways to make Jackson lenient, from most local to most global:</p>

<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// 1. Per DTO/type - the local fix:</span>
<span class="nd">@JsonIgnoreProperties</span><span class="o">(</span><span class="n">ignoreUnknown</span> <span class="o">=</span> <span class="kc">true</span><span class="o">)</span>
<span class="kd">public</span> <span class="kd">record</span> <span class="nf">Room</span><span class="o">(</span><span class="nc">String</span> <span class="n">id</span><span class="o">,</span> <span class="nc">String</span> <span class="n">name</span><span class="o">,</span> <span class="kt">int</span> <span class="n">capacity</span><span class="o">)</span> <span class="o">{</span> <span class="o">}</span>
</code></pre></div></div>

<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// 2. Per ObjectMapper - the application-wide fix:</span>
<span class="nc">ObjectMapper</span> <span class="n">mapper</span> <span class="o">=</span> <span class="nc">JsonMapper</span><span class="o">.</span><span class="na">builder</span><span class="o">()</span>
        <span class="o">.</span><span class="na">disable</span><span class="o">(</span><span class="nc">DeserializationFeature</span><span class="o">.</span><span class="na">FAIL_ON_UNKNOWN_PROPERTIES</span><span class="o">)</span>
        <span class="o">.</span><span class="na">build</span><span class="o">();</span>
</code></pre></div></div>

<p>For the MicroProfile REST Client specifically, you&#8217;d expose that configured <code class="language-plaintext highlighter-rouge">ObjectMapper</code> through a <code class="language-plaintext highlighter-rouge">ContextResolver&lt;ObjectMapper&gt;</code> so the client picks it up:</p>

<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// 3. Hand the configured mapper to the REST Client via a ContextResolver:</span>
<span class="nd">@Provider</span>
<span class="kd">public</span> <span class="kd">class</span> <span class="nc">LenientJacksonProvider</span> <span class="kd">implements</span> <span class="nc">ContextResolver</span><span class="o">&lt;</span><span class="nc">ObjectMapper</span><span class="o">&gt;</span> <span class="o">{</span>

    <span class="kd">private</span> <span class="kd">final</span> <span class="nc">ObjectMapper</span> <span class="n">mapper</span> <span class="o">=</span> <span class="nc">JsonMapper</span><span class="o">.</span><span class="na">builder</span><span class="o">()</span>
            <span class="o">.</span><span class="na">disable</span><span class="o">(</span><span class="nc">DeserializationFeature</span><span class="o">.</span><span class="na">FAIL_ON_UNKNOWN_PROPERTIES</span><span class="o">)</span>
            <span class="o">.</span><span class="na">build</span><span class="o">();</span>

    <span class="nd">@Override</span>
    <span class="kd">public</span> <span class="nc">ObjectMapper</span> <span class="nf">getContext</span><span class="o">(</span><span class="nc">Class</span><span class="o">&lt;?&gt;</span> <span class="n">type</span><span class="o">)</span> <span class="o">{</span>
        <span class="k">return</span> <span class="n">mapper</span><span class="o">;</span>
    <span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>

<p>In a real MicroProfile REST Client, make sure this provider is actually registered with the client, for example with <code class="language-plaintext highlighter-rouge">@RegisterProvider</code>, MicroProfile REST Client configuration, or your runtime&#8217;s provider discovery mechanism.</p>

<p>The <a href="https://fasterxml.github.io/jackson-annotations/javadoc/2.18/com/fasterxml/jackson/annotation/JsonIgnoreProperties.html"><code class="language-plaintext highlighter-rouge">@JsonIgnoreProperties(ignoreUnknown = true)</code></a> annotation is the one most people reach for first, because it is right next to the DTO and easy to read.</p>

<h3 id="how-to-call-it-1">How to call it</h3>

<p>The HTTP call is identical &#8211; the difference is entirely server-payload vs. client-config. With strict Jackson and the six-field payload, the response is still valid JSON, but a Jackson-backed client trying to deserialize it into the three-field <code class="language-plaintext highlighter-rouge">Room</code> now fails:</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>curl <span class="nt">-X</span> GET http://localhost:8080/api/unknown-fields/room-7 <span class="se">\</span>
  <span class="nt">-H</span> <span class="s2">"Accept: application/json"</span>
</code></pre></div></div>

<p><strong>&#9989; Pros:</strong></p>
<ul>
  <li>Catches typos and contract drift early &#8211; a renamed field shows up as a loud failure, not silent data loss.</li>
  <li>Explicit: you opt in to every field you accept.</li>
</ul>

<p><strong>&#10060; Cons:</strong></p>
<ul>
  <li>An additive, backwards-compatible server change breaks your client. This is the one that surprises people.</li>
  <li>The fix lives in client code/config, which means you can&#8217;t always fix it quickly if you don&#8217;t own the client.</li>
</ul>

<p>&#9888;&#65039; <em><strong>Caution:</strong> If you consume third-party APIs with raw Jackson defaults, you are one additive change away from an incident. Either set <code class="language-plaintext highlighter-rouge">@JsonIgnoreProperties(ignoreUnknown = true)</code> on your DTOs, or disable <code class="language-plaintext highlighter-rouge">FAIL_ON_UNKNOWN_PROPERTIES</code> globally &#8211; and do it deliberately, not by accident.</em></p>

<hr />

<h2 id="3-quarkus-jackson-but-lenient">3. Quarkus: Jackson, but lenient</h2>

<h3 id="what-it-looks-like-2">What it looks like</h3>

<p>Here is where it gets interesting, and where a lot of confusion comes from. Quarkus uses Jackson for a great deal of its JSON handling &#8211; but it does <strong>not</strong> keep Jackson&#8217;s strict default.</p>

<p>Quarkus ships with <code class="language-plaintext highlighter-rouge">quarkus.jackson.fail-on-unknown-properties=false</code> as its <strong>default</strong>, which means a Quarkus app with Jackson <strong>ignores unknown properties out of the box</strong> &#8211; the opposite of what you&#8217;d get from a bare <code class="language-plaintext highlighter-rouge">ObjectMapper</code>. This is documented in the <a href="https://quarkus.io/guides/all-config#quarkus-jackson_quarkus-jackson-fail-on-unknown-properties">Quarkus Jackson configuration reference</a> and the <a href="https://quarkus.io/guides/rest-json">Quarkus JSON guide</a>.</p>

<p>So the same Jackson library behaves differently depending on whether Quarkus configured it for you or you <code class="language-plaintext highlighter-rouge">new</code>-ed up an <code class="language-plaintext highlighter-rouge">ObjectMapper</code> yourself. If you want the strict behaviour back, you flip the property:</p>

<div class="language-properties highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># application.properties - opt back in to strict deserialization
</span><span class="py">quarkus.jackson.fail-on-unknown-properties</span><span class="p">=</span><span class="s">true</span>
</code></pre></div></div>

<p>&#8230;or override it for a single class with the same <code class="language-plaintext highlighter-rouge">@JsonIgnoreProperties</code> annotation from &#167;2.</p>

<h3 id="how-to-call-it-2">How to call it</h3>

<div class="language-http highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="err">GET http://localhost:8080/api/unknown-fields/room-7
Accept: application/json
</span></code></pre></div></div>

<p>On Quarkus defaults, this succeeds even with the six-field payload, because Quarkus pre-configured Jackson to be lenient.</p>

<p><strong>&#9989; Pros:</strong></p>
<ul>
  <li>Sensible &#8220;tolerant reader&#8221; default for a consumer, even though the underlying library is Jackson.</li>
  <li>One property toggles the behaviour for the whole app.</li>
</ul>

<p><strong>&#10060; Cons:</strong></p>
<ul>
  <li>It diverges from &#8220;stock Jackson&#8221;, which trips up anyone who knows Jackson&#8217;s default and assumes it applies here.</li>
  <li>Behaviour now depends on <em>where</em> the <code class="language-plaintext highlighter-rouge">ObjectMapper</code> comes from (Quarkus-managed vs. hand-rolled).</li>
</ul>

<p>&#129514; <em><strong>Observation:</strong> This is a perfect example of why &#8220;we use Jackson&#8221; is not enough information. The framework around Jackson decides the default, and Quarkus and a plain <code class="language-plaintext highlighter-rouge">ObjectMapper</code> land on opposite answers.</em></p>

<hr />

<h2 id="4-spring-boot-also-lenient">4. Spring Boot: also lenient</h2>

<h3 id="what-it-looks-like-3">What it looks like</h3>

<p>Spring Boot ends up in the same place as Quarkus: it uses Jackson, but configures it to be lenient by default. Stock Jackson is strict, but in the current <a href="https://docs.spring.io/spring-boot/4.1.0/reference/features/json.html">Spring Boot 4.1 reference documentation</a>, <code class="language-plaintext highlighter-rouge">spring.jackson.deserialization.fail-on-unknown-properties=false</code> is the documented default, so Spring also <strong>ignores</strong> unknown fields out of the box.</p>

<div class="language-properties highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># application.properties - flip Spring back to strict if you want it
</span><span class="py">spring.jackson.deserialization.fail-on-unknown-properties</span><span class="p">=</span><span class="s">true</span>
</code></pre></div></div>

<p>So if you&#8217;re coming from Spring, the surprise is similar to Quarkus: you are using Jackson, but not stock Jackson defaults.</p>

<hr />

<h2 id="a-note-on-the-other-direction-server-receiving-extra-fields">A note on the other direction (server receiving extra fields)</h2>

<p>So far we&#8217;ve looked at the <em>client</em> receiving more than it expected. The mirror image is your <em>server</em> receiving a request body with extra fields &#8211; a client POSTs more than your endpoint&#8217;s DTO declares. The good news is that it is the <strong>same providers and the same switches</strong>:</p>

<ul>
  <li>On <strong>JSON-B / Yasson</strong>, the extra fields in the inbound body are ignored by default.</li>
  <li>On <strong>raw Jackson</strong>, the inbound body fails with <code class="language-plaintext highlighter-rouge">UnrecognizedPropertyException</code> unless you set <code class="language-plaintext highlighter-rouge">@JsonIgnoreProperties(ignoreUnknown = true)</code> or disable <code class="language-plaintext highlighter-rouge">FAIL_ON_UNKNOWN_PROPERTIES</code>.</li>
  <li>On <strong>Quarkus</strong> and <strong>Spring Boot</strong>, the extra fields are ignored by default, for the same reasons as above.</li>
</ul>

<p>There is one extra wrinkle worth flagging on the server side: silently ignoring unknown fields on <em>input</em> can be a mild security/robustness smell. A client sending fields you don&#8217;t recognise might be confused, might be on the wrong API version, or might be probing. Strictness on input is sometimes a feature, not a bug. This is the opposite of what is expected for a consumer.</p>

<blockquote>
  <p>&#128161; <em><strong>Note:</strong> The mental model is &#8220;tolerant on the way in is convenient, strict on the way in is defensive&#8221;. You get to choose per endpoint &#8211; just choose on purpose.</em></p>
</blockquote>

<hr />

<h2 id="summary-comparison">Summary Comparison</h2>

<p>The table below assumes the six-field JSON payload from the introduction being mapped onto the three-field <code class="language-plaintext highlighter-rouge">Room</code> DTO.</p>

<table class="bordered-table">
  <thead>
    <tr>
      <th>Provider / stack</th>
      <th>Default on unknown fields</th>
      <th>Result with extra fields</th>
      <th>How to flip it</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td><strong>JSON-B / Yasson</strong></td>
      <td>Ignore</td>
      <td>&#9989; Deserializes, extra fields dropped</td>
      <td>(Already lenient; strictness needs custom validation/deserialization logic)</td>
    </tr>
    <tr>
      <td><strong>Jackson (stock)</strong></td>
      <td><strong>Fail</strong></td>
      <td>&#10060; <code class="language-plaintext highlighter-rouge">UnrecognizedPropertyException</code></td>
      <td><code class="language-plaintext highlighter-rouge">@JsonIgnoreProperties(ignoreUnknown=true)</code> or disable the feature</td>
    </tr>
    <tr>
      <td><strong>Quarkus + Jackson</strong></td>
      <td>Ignore</td>
      <td>&#9989; Deserializes, extra fields dropped</td>
      <td><code class="language-plaintext highlighter-rouge">quarkus.jackson.fail-on-unknown-properties=true</code></td>
    </tr>
    <tr>
      <td><strong>Spring Boot + Jackson</strong></td>
      <td>Ignore</td>
      <td>&#9989; Deserializes, extra fields dropped</td>
      <td><code class="language-plaintext highlighter-rouge">spring.jackson.deserialization.fail-on-unknown-properties=true</code></td>
    </tr>
  </tbody>
</table>

<p>The one row that catches people out is <strong>Jackson (stock)</strong> &#8211; and by extension any MicroProfile REST Client where you added a bare Jackson provider without configuring it.</p>

<h2 id="conclusion">Conclusion</h2>

<p>The MicroProfile REST Client doesn&#8217;t have an opinion on unknown fields &#8211; it hands that decision to your JSON provider, and the providers don&#8217;t agree:</p>

<ul>
  <li><strong>JSON-B / Yasson</strong> ignores them, by spec.</li>
  <li><strong>Stock Jackson</strong> fails, by its own default.</li>
  <li><strong>Quarkus and Spring Boot</strong> both use Jackson but pre-configure it to ignore them.</li>
</ul>

<p>So the practical advice is short. If you&#8217;re writing a consumer, you almost certainly want the tolerant-reader behaviour, so additive changes on the server don&#8217;t page you at 2am. On JSON-B you already have it. On raw Jackson, add <code class="language-plaintext highlighter-rouge">@JsonIgnoreProperties(ignoreUnknown = true)</code> to your DTOs (or disable <code class="language-plaintext highlighter-rouge">FAIL_ON_UNKNOWN_PROPERTIES</code> once, globally) and be explicit that you&#8217;ve made that choice. And whatever you do, know which default you&#8217;re actually running &#8211; because &#8220;we use Jackson&#8221; tells you almost nothing until you also know what configured it.</p>

<h2 id="whats-next">What&#8217;s Next?</h2>

<p>Unknown fields are one small corner of building REST clients and APIs that survive change. The same <a href="https://github.com/mehmandarov/api-guide-java">API Guide for Java</a> repo covers OpenAPI documentation, error handling, security, pagination, and versioning &#8211; see my earlier posts on <a href="/api-versioning/">API versioning in Java using JAX-RS</a> and <a href="/rfc-9457-problem-details-jakarta-ee/">RFC 9457 Problem Details</a>. The next post in this little run looks at the opposite of returning a <code class="language-plaintext highlighter-rouge">byte[]</code>: building an endpoint that <em>accepts</em> binary attachments as part of the payload.</p>

<p><strong><em>Happy (and tolerant) reading, folks!</em></strong></p>

<hr />]]></content><author><name>Rustam Mehmandarov</name></author><summary type="html">What happens when a JSON response contains more fields than your DTO declares? The answer depends on your JSON provider. A look at the defaults for JSON-B, Jackson, Quarkus, and Spring when consuming an API with the MicroProfile REST Client &#8211; with the official docs to back each one up.</summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://mehmandarov.com/assets/images/posts-images/computers.jpg"/><category term="blog"/><category term="english"/><category term="java"/><category term="api"/><category term="jakarta ee"/><category term="microprofile"/><category term="jax-rs"/><category term="quarkus"/><category term="rest client"/><category term="spring boot"/></entry><entry><title type="html">Jakarta EL: safer rules without handwritten predicate parsers</title><link href="https://mehmandarov.com/jakarta-el-safer-rules-without-handwritten-predicate-parsers/" rel="alternate" type="text/html" title="Jakarta EL: safer rules without handwritten predicate parsers"/><published>2026-06-17T16:17:00+02:00</published><updated>2026-06-17T16:17:00+02:00</updated><id>https://mehmandarov.com/jakarta-el-safer-rules-without-handwritten-predicate-parsers</id><content type="html" xml:base="https://mehmandarov.com/jakarta-el-safer-rules-without-handwritten-predicate-parsers/"><![CDATA[<p><em><a href="https://jakarta.ee/specifications/expression-language/">Jakarta Expression Language</a> (EL) is already part of the platform, but we usually meet it indirectly through Faces, CDI, or Bean Validation. In this post, we will use it directly: as a small in-process predicate engine for authorization policies. We will replace the handwritten parser, keep the same policy file format, and then look at what it takes to make evaluation safe.</em></p>

<ul>
  <li><a href="#introduction">Introduction</a></li>
  <li><a href="#1-why-jakarta-el">1. Why Jakarta EL?</a>
    <ul>
      <li><a href="#11-from-handwritten-predicate-parser-to-el">1.1 From handwritten predicate parser to EL</a></li>
    </ul>
  </li>
  <li><a href="#2-we-need-to-talk-about-the-security">2. We need to talk about the security</a>
    <ul>
      <li><a href="#21-how-dangerous-can-it-be-why-bother-validating-the-input">2.1 How dangerous can it be? Why bother validating the input?</a></li>
      <li><a href="#22-adding-security-to-the-expressiveness">2.2 Adding security to the expressiveness</a></li>
      <li><a href="#23-replacing-the-parserevaluator-with-jakarta-el">2.3 Replacing the parser/evaluator with Jakarta EL</a></li>
    </ul>
  </li>
  <li><a href="#3-trade-offs-and-when-to-use-what">3. Trade-offs and when to use what</a>
    <ul>
      <li><a href="#31-what-ships-today-vs-where-were-headed">3.1 What ships today vs. where we&#8217;re headed</a></li>
      <li><a href="#32-is-the-swap-worth-it">3.2 Is the swap worth it?</a></li>
      <li><a href="#33-summary-comparison">3.3 Summary comparison</a></li>
    </ul>
  </li>
  <li><a href="#conclusion">Conclusion</a></li>
  <li><a href="#whats-next">What&#8217;s Next?</a></li>
</ul>

<hr />

<h2 id="introduction">Introduction</h2>

<p>If you&#8217;ve written Jakarta EE for any length of time, you&#8217;ve used Jakarta Expression Language (EL) whether you noticed it or not. Every <code class="language-plaintext highlighter-rouge">#{bean.name}</code> in a Faces (JSF) page, and every <code class="language-plaintext highlighter-rouge">${validatedValue}</code> in a Bean Validation constraint message, is EL under the hood &#8211; a small expression language sitting underneath much of the platform.</p>

<p>Quick terminology note before we go further: in this post, a <strong>predicate</strong> is just the boolean condition part of a rule &#8211; the bit that answers &#8220;does this speaker get to see this event?&#8221;. The whole JSON object is the rule; the <code class="language-plaintext highlighter-rouge">predicate</code> field is the expression inside that rule.</p>

<p>What EL is genuinely good at is being a JavaBean-aware predicate language inside your own app. Because expressions are just strings, EL also fits neatly into policy files that your app already hot-reloads. You don&#8217;t have to write a parser, you don&#8217;t have to pull in dependencies like Drools or OPA &#8211; <code class="language-plaintext highlighter-rouge">jakarta.el.ELProcessor</code> is part of Jakarta Expression Language, which ships with Jakarta EE runtimes, and you can call it directly from your Java code.</p>

<p>If you have heard of Jakarta EL, you might have also heard that EL has a reputation for danger. However, that reputation is mostly about untrusted or insufficiently validated input reaching an evaluator with too much access &#8211; not about EL being uniquely dangerous by itself. The handful of CVEs behind the reputation were patched years ago in supported versions, but the lesson behind them still applies to any string you &#8220;evaluate&#8221; in any language or framework.</p>

<p>We&#8217;ll build something useful with EL first. Later, we will look at the potential security issues and their mitigation, and how you can make it safe by restricting what EL can access with a custom <code class="language-plaintext highlighter-rouge">ELResolver</code>.</p>

<h2 id="1-why-jakarta-el">1. Why Jakarta EL?</h2>

<p>Surprisingly often, people end up implementing and re-implementing small DSL parsers for policy expressions. Oftentimes, it turns out to be a subset of what Jakarta EL can offer. EL gives you the following, for free, out of the box:</p>

<ul>
  <li><strong>JavaBean-style and <code class="language-plaintext highlighter-rouge">Map</code>-backed property access</strong>. EL natively understands standard <code class="language-plaintext highlighter-rouge">getX()</code> getters, and resolves <code class="language-plaintext highlighter-rouge">a.b</code> against <code class="language-plaintext highlighter-rouge">Map</code> keys just as happily. In this demo we flatten <code class="language-plaintext highlighter-rouge">speaker</code> and <code class="language-plaintext highlighter-rouge">event</code> into small per-call maps before evaluation, so <code class="language-plaintext highlighter-rouge">speaker.languages</code> or <code class="language-plaintext highlighter-rouge">event.cfpDeadline</code> resolves a map entry &#8211; no extra mapping code, and no domain record methods left exposed to the expression.</li>
  <li><strong>Method calls</strong> on the object graph. You can also call methods on what you <em>do</em> expose &#8211; e.g. <code class="language-plaintext highlighter-rouge">event.cfpDeadline.isAfter(...)</code> on a <code class="language-plaintext highlighter-rouge">LocalDate</code> value.</li>
  <li><strong>Standard arithmetic and boolean operators</strong>. You can use <code class="language-plaintext highlighter-rouge">&amp;&amp;</code>, <code class="language-plaintext highlighter-rouge">||</code>, <code class="language-plaintext highlighter-rouge">==</code>, <code class="language-plaintext highlighter-rouge">!=</code>, <code class="language-plaintext highlighter-rouge">&lt;</code>, <code class="language-plaintext highlighter-rouge">&gt;</code>, <code class="language-plaintext highlighter-rouge">+</code>, <code class="language-plaintext highlighter-rouge">-</code>, <code class="language-plaintext highlighter-rouge">*</code>, <code class="language-plaintext highlighter-rouge">/</code>, <code class="language-plaintext highlighter-rouge">%</code> out of the box.</li>
  <li><strong><code class="language-plaintext highlighter-rouge">empty</code>, <code class="language-plaintext highlighter-rouge">not</code>, and ternary <code class="language-plaintext highlighter-rouge">? :</code></strong> for shorter and more flexible expressions.</li>
  <li>A pluggable <a href="https://jakarta.ee/specifications/expression-language/6.0/apidocs/jakarta.el/jakarta/el/elresolver"><code class="language-plaintext highlighter-rouge">ELResolver</code></a> chain that lets you <strong>decide what is and isn&#8217;t reachable</strong> from an expression.</li>
</ul>

<p><strong>Important note:</strong> The last bullet point is the one this post is <em>really</em> about. Raw EL evaluation can be dangerous if the resolver chain exposes too much: <code class="language-plaintext highlighter-rouge">getClass()</code>, reflection, runtime classes, and other things you do not want policy expressions to reach. EL becomes <em>safer</em> once you restrict what the resolver chain can access.</p>

<hr />

<h3 id="11-from-handwritten-predicate-parser-to-el">1.1 From handwritten predicate parser to EL</h3>

<p>Let&#8217;s look at an example from gem #2 of my <a href="https://github.com/mehmandarov/jakarta-hidden-gems">Jakarta EE Hidden Gems</a> demo code. There, authorization rules live in a <a href="https://github.com/mehmandarov/jakarta-hidden-gems/blob/main/policy/rules.json">JSON policy file</a>. Gem #2 used a handwritten predicate parser/evaluator; this post &#8211; gem #3 &#8211; keeps the same predicates and the same <code class="language-plaintext highlighter-rouge">rules.json</code> format, but swaps the evaluator behind them for an EL-based one. The enforcement path still goes through the same <a href="https://github.com/mehmandarov/jakarta-hidden-gems/blob/main/02-authorization/src/main/java/com/mehmandarov/jhg/authz/AccessPolicy.java"><code class="language-plaintext highlighter-rouge">@RolesAllowed</code> + JAX-RS ABAC filter</a>:</p>

<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w">
  </span><span class="nl">"rules"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
    </span><span class="p">{</span><span class="w">
      </span><span class="nl">"subject"</span><span class="p">:</span><span class="w">   </span><span class="s2">"role:SPEAKER"</span><span class="p">,</span><span class="w">
      </span><span class="nl">"action"</span><span class="p">:</span><span class="w">    </span><span class="s2">"GET"</span><span class="p">,</span><span class="w">
      </span><span class="nl">"resource"</span><span class="p">:</span><span class="w">  </span><span class="s2">"/api/events/*"</span><span class="p">,</span><span class="w">
      </span><span class="nl">"predicate"</span><span class="p">:</span><span class="w"> </span><span class="s2">"speaker.languages intersects event.languages &amp;&amp; speaker.tracks intersects event.tracks"</span><span class="w">
    </span><span class="p">}</span><span class="w">
  </span><span class="p">]</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>

<p>Here, each rule carries a <code class="language-plaintext highlighter-rouge">predicate</code> string, evaluated by a tiny <a href="https://github.com/mehmandarov/jakarta-hidden-gems/blob/main/02-authorization/src/main/java/com/mehmandarov/jhg/authz/PredicateEvaluator.java">handwritten parser</a> &#8211; deliberately limited to four operators (<code class="language-plaintext highlighter-rouge">intersects</code>, <code class="language-plaintext highlighter-rouge">contains</code>, <code class="language-plaintext highlighter-rouge">==</code>, <code class="language-plaintext highlighter-rouge">!=</code>) joined only by a top-level <code class="language-plaintext highlighter-rouge">&amp;&amp;</code>, with no nesting, method calls, or functions. Even at that size, the implementation is more complex than you&#8217;d expect. Here&#8217;s just the reference-resolution helper (note: the whole file is linked above):</p>

<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">private</span> <span class="kd">static</span> <span class="nc">Object</span> <span class="nf">resolve</span><span class="o">(</span><span class="nc">String</span> <span class="n">ref</span><span class="o">,</span> <span class="nc">Map</span><span class="o">&lt;</span><span class="nc">String</span><span class="o">,</span> <span class="nc">Object</span><span class="o">&gt;</span> <span class="n">ctx</span><span class="o">)</span> <span class="o">{</span>
    <span class="k">if</span> <span class="o">(</span><span class="n">ref</span><span class="o">.</span><span class="na">isEmpty</span><span class="o">())</span> <span class="k">return</span> <span class="kc">null</span><span class="o">;</span>
    <span class="kt">char</span> <span class="n">c0</span> <span class="o">=</span> <span class="n">ref</span><span class="o">.</span><span class="na">charAt</span><span class="o">(</span><span class="mi">0</span><span class="o">);</span>

    <span class="k">if</span> <span class="o">(</span><span class="n">c0</span> <span class="o">==</span> <span class="sc">'"'</span> <span class="o">||</span> <span class="n">c0</span> <span class="o">==</span> <span class="sc">'\''</span><span class="o">)</span> <span class="o">{</span>
        <span class="k">return</span> <span class="n">ref</span><span class="o">.</span><span class="na">substring</span><span class="o">(</span><span class="mi">1</span><span class="o">,</span> <span class="n">ref</span><span class="o">.</span><span class="na">length</span><span class="o">()</span> <span class="o">-</span> <span class="mi">1</span><span class="o">);</span>
    <span class="o">}</span>
    <span class="k">if</span> <span class="o">(</span><span class="nc">Character</span><span class="o">.</span><span class="na">isDigit</span><span class="o">(</span><span class="n">c0</span><span class="o">)</span> <span class="o">||</span> <span class="n">c0</span> <span class="o">==</span> <span class="sc">'-'</span><span class="o">)</span> <span class="o">{</span>
        <span class="k">try</span> <span class="o">{</span> <span class="k">return</span> <span class="nc">Long</span><span class="o">.</span><span class="na">parseLong</span><span class="o">(</span><span class="n">ref</span><span class="o">);</span> <span class="o">}</span> <span class="k">catch</span> <span class="o">(</span><span class="nc">NumberFormatException</span> <span class="n">ignored</span><span class="o">)</span> <span class="o">{}</span>
        <span class="k">try</span> <span class="o">{</span> <span class="k">return</span> <span class="nc">Double</span><span class="o">.</span><span class="na">parseDouble</span><span class="o">(</span><span class="n">ref</span><span class="o">);</span> <span class="o">}</span> <span class="k">catch</span> <span class="o">(</span><span class="nc">NumberFormatException</span> <span class="n">ignored</span><span class="o">)</span> <span class="o">{}</span>
    <span class="o">}</span>
    <span class="k">return</span> <span class="nf">switch</span> <span class="o">(</span><span class="n">ref</span><span class="o">)</span> <span class="o">{</span>
        <span class="k">case</span> <span class="s">"true"</span>  <span class="o">-&gt;</span> <span class="nc">Boolean</span><span class="o">.</span><span class="na">TRUE</span><span class="o">;</span>
        <span class="k">case</span> <span class="s">"false"</span> <span class="o">-&gt;</span> <span class="nc">Boolean</span><span class="o">.</span><span class="na">FALSE</span><span class="o">;</span>
        <span class="k">case</span> <span class="s">"null"</span>  <span class="o">-&gt;</span> <span class="kc">null</span><span class="o">;</span>
        <span class="k">default</span> <span class="o">-&gt;</span> <span class="o">{</span>
            <span class="nc">String</span><span class="o">[]</span> <span class="n">segments</span> <span class="o">=</span> <span class="n">ref</span><span class="o">.</span><span class="na">split</span><span class="o">(</span><span class="s">"\\."</span><span class="o">);</span>
            <span class="nc">Object</span> <span class="n">cur</span> <span class="o">=</span> <span class="n">ctx</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="n">segments</span><span class="o">[</span><span class="mi">0</span><span class="o">]);</span>
            <span class="k">for</span> <span class="o">(</span><span class="kt">int</span> <span class="n">i</span> <span class="o">=</span> <span class="mi">1</span><span class="o">;</span> <span class="n">i</span> <span class="o">&lt;</span> <span class="n">segments</span><span class="o">.</span><span class="na">length</span> <span class="o">&amp;&amp;</span> <span class="n">cur</span> <span class="o">!=</span> <span class="kc">null</span><span class="o">;</span> <span class="n">i</span><span class="o">++)</span> <span class="o">{</span>
                <span class="n">cur</span> <span class="o">=</span> <span class="n">property</span><span class="o">(</span><span class="n">cur</span><span class="o">,</span> <span class="n">segments</span><span class="o">[</span><span class="n">i</span><span class="o">]);</span>
            <span class="o">}</span>
            <span class="n">yield</span> <span class="n">cur</span><span class="o">;</span>
        <span class="o">}</span>
    <span class="o">};</span>
<span class="o">}</span>
</code></pre></div></div>

<p>And that&#8217;s just one of several methods: a top-level <code class="language-plaintext highlighter-rouge">&amp;&amp;</code> splitter, a per-clause evaluator, an operator enum with collection-aware <code class="language-plaintext highlighter-rouge">intersects</code>/<code class="language-plaintext highlighter-rouge">contains</code>, and JavaBean/record reflection hiding inside <code class="language-plaintext highlighter-rouge">property(...)</code>. About 160 lines total, and every line has to grow if a policy author asks for <code class="language-plaintext highlighter-rouge">||</code>, dates, or any function call.</p>

<p>Now, what if we swap the evaluator with a <a href="https://jakarta.ee/specifications/expression-language/6.0/apidocs/jakarta.el/jakarta/el/elprocessor"><code class="language-plaintext highlighter-rouge">jakarta.el.ELProcessor</code></a> while keeping the same <code class="language-plaintext highlighter-rouge">rules.json</code> format? That gives us room for rules the old parser could not express:</p>

<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w">
  </span><span class="nl">"rules"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
    </span><span class="p">{</span><span class="w">
      </span><span class="nl">"subject"</span><span class="p">:</span><span class="w">   </span><span class="s2">"role:SPEAKER"</span><span class="p">,</span><span class="w">
      </span><span class="nl">"action"</span><span class="p">:</span><span class="w">    </span><span class="s2">"GET"</span><span class="p">,</span><span class="w">
      </span><span class="nl">"resource"</span><span class="p">:</span><span class="w">  </span><span class="s2">"/api/events/*"</span><span class="p">,</span><span class="w">
      </span><span class="nl">"predicate"</span><span class="p">:</span><span class="w"> </span><span class="s2">"speaker.languages intersects event.languages &amp;&amp; fn.daysUntil(event.cfpDeadline) &gt; 7"</span><span class="w">
    </span><span class="p">}</span><span class="w">
  </span><span class="p">]</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>

<p>And the EL-based evaluator itself is actually shorter than the handwritten parser it replaces (as we&#8217;ll see in the <a href="https://github.com/mehmandarov/jakarta-hidden-gems/blob/main/03-el-policy/src/main/java/com/mehmandarov/jhg/el/ElPredicateEngine.java"><code class="language-plaintext highlighter-rouge">ElPredicateEngine</code></a> code below). Note that the policy syntax stays unchanged. Gem #3 rewrites <code class="language-plaintext highlighter-rouge">intersects</code> into a helper-function call, so old rules still work while new rules can use things like <em>date arithmetic</em> through the <code class="language-plaintext highlighter-rouge">fn</code> helper bean. As a bonus, this fits the existing policy reload mechanism &#8211; edit the policy file, let the <code class="language-plaintext highlighter-rouge">WatchService</code> reload it, and the new predicate is in effect without a redeploy.</p>

<p>There is a catch, though. If you can <em>evaluate</em> almost anything, you can <em>run</em> almost anything. You might think, &#8220;So what?&#8221; Well, consider one of the seemingly innocent tools at your disposal: <code class="language-plaintext highlighter-rouge">getClass()</code>. In permissive resolver setups, exposing arbitrary object graphs can let an expression walk from <code class="language-plaintext highlighter-rouge">getClass()</code> into reflection APIs. Written as plain Java, the kind of chain you are trying to prevent looks like this:</p>

<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// The reflection chain reachable from getClass(), written out in Java.</span>
<span class="c1">// Reaching System.exit(...) tears down the whole JVM, and the app server with it.</span>
<span class="s">""</span><span class="o">.</span><span class="na">getClass</span><span class="o">().</span><span class="na">forName</span><span class="o">(</span><span class="s">"java.lang.System"</span><span class="o">)</span>
   <span class="o">.</span><span class="na">getMethod</span><span class="o">(</span><span class="s">"exit"</span><span class="o">,</span> <span class="kt">int</span><span class="o">.</span><span class="na">class</span><span class="o">)</span>
   <span class="o">.</span><span class="na">invoke</span><span class="o">(</span><span class="kc">null</span><span class="o">,</span> <span class="mi">1</span><span class="o">);</span>
</code></pre></div></div>

<h2 id="2-we-need-to-talk-about-the-security">2. We need to talk about the security</h2>

<h3 id="21-how-dangerous-can-it-be-why-bother-validating-the-input">2.1 How dangerous can it be? Why bother validating the input?</h3>

<p>EL&#8217;s reputation comes from a handful of CVEs, and they&#8217;re worth knowing &#8211; not because current patched versions should still behave that way, but because they share one shape: <strong>a string the application treated as a label or a template was actually an EL expression, built from unvalidated user input.</strong></p>

<p>The textbook example is three lines of Bean Validation that shipped in tutorials all over the internet:</p>

<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">public</span> <span class="kd">class</span> <span class="nc">CreateTalkRequest</span> <span class="o">{</span>
    <span class="nd">@Size</span><span class="o">(</span><span class="n">max</span> <span class="o">=</span> <span class="mi">10</span><span class="o">,</span> <span class="n">message</span> <span class="o">=</span> <span class="s">"${validatedValue} is too long"</span><span class="o">)</span>
    <span class="kd">private</span> <span class="nc">String</span> <span class="n">title</span><span class="o">;</span>
<span class="o">}</span>
</code></pre></div></div>

<p>Looks innocent. It is &#8220;just&#8221; an error message, right? But <code class="language-plaintext highlighter-rouge">validatedValue</code> is the user-provided value (<code class="language-plaintext highlighter-rouge">title</code> in this case), and Bean Validation message templates go through an interpolator that understands EL. The important nuance is that the constant template above is not, by itself, the whole vulnerability story in modern patched providers. The dangerous shape is when user-controlled text is allowed to become part of the message template, or to escape into template evaluation, instead of being treated as plain text. With the wrong combination of code and provider version, that meant <strong>remote code execution via a validation error message</strong>. That was the shape of <a href="https://nvd.nist.gov/vuln/detail/CVE-2020-10693">CVE-2020-10693</a> in Hibernate Validator &#8211; fixed back in 2020 by interpolating against a constant template.</p>

<p>Two that are worth recognising by name, both long since patched:</p>

<ul>
  <li><strong><a href="https://nvd.nist.gov/vuln/detail/CVE-2020-10693">CVE-2020-10693</a></strong> &#8211; Hibernate Validator: user input passed straight into a constraint-message template (via an interpolation bypass).</li>
  <li><strong><a href="https://nvd.nist.gov/vuln/detail/CVE-2017-1000486">CVE-2017-1000486</a></strong> &#8211; PrimeFaces: an encrypted JSF parameter intended to hold EL was compromised due to a weak default key, allowing attackers to forge and execute arbitrary EL expressions on the server.</li>
</ul>

<p>And to be clear, this is not a problem unique to Jakarta EL. Other expression languages face the same risk when evaluated against untrusted input. For example, <strong><a href="https://nvd.nist.gov/vuln/detail/CVE-2018-1273">CVE-2018-1273</a></strong> was a very similar RCE in Spring Data, but it happened via SpEL (Spring Expression Language), not Jakarta EL.</p>

<p>The lesson is <em>not</em> &#8220;avoid EL&#8221;. It&#8217;s the same lesson as SQL injection, command injection, and every other injection class: <strong>never feed unvalidated input to an evaluator.</strong> The <code class="language-plaintext highlighter-rouge">SafeELEvaluator</code> is exactly how you act on that lesson &#8211; the predicates come from a trusted policy file, the resolver whitelist decides what&#8217;s reachable, and a watchdog caps execution time.</p>

<p><em><strong>Caution:</strong> If you ever find yourself writing <code class="language-plaintext highlighter-rouge">String message = "..." + userInput + "..."</code> and then handing the result to anything called <code class="language-plaintext highlighter-rouge">interpolate</code>, <code class="language-plaintext highlighter-rouge">evaluate</code>, or <code class="language-plaintext highlighter-rouge">process</code>, stop. That&#8217;s the shape every one of the CVEs above has in common &#8211; and it&#8217;s a property of the input, not of EL.</em></p>

<h3 id="22-adding-security-to-the-expressiveness">2.2 Adding security to the expressiveness</h3>

<p>So how do you get EL&#8217;s expressiveness without the danger? In the repo there are two EL-based variants. <a href="https://github.com/mehmandarov/jakarta-hidden-gems/blob/main/03-el-policy/src/main/java/com/mehmandarov/jhg/el/ElPredicateEngine.java"><code class="language-plaintext highlighter-rouge">ElPredicateEngine</code></a> is the pragmatic gem #3 default: it keeps gem #2&#8217;s policy syntax, rewrites <code class="language-plaintext highlighter-rouge">intersects</code> and <code class="language-plaintext highlighter-rouge">contains</code> to helper calls, and uses a fail-closed denylist. <a href="https://github.com/mehmandarov/jakarta-hidden-gems/blob/main/03-el-policy/src/main/java/com/mehmandarov/jhg/el/SafeELEvaluator.java"><code class="language-plaintext highlighter-rouge">SafeELEvaluator</code></a> is the hardened version: pure EL, a type-whitelisting resolver, a resolution budget, and a timeout. The first one shows the migration path; the second one is the shape you want when you care about the sandbox.</p>

<p>You&#8217;ve seen that raw EL evaluation can be permissive, and a default resolver chain may resolve things like <code class="language-plaintext highlighter-rouge">getClass()</code> if you expose the wrong object graph. When you want EL purely as a predicate language, the fix is to <strong>add a restricted resolver</strong> that refuses to reach those types in the first place. With <code class="language-plaintext highlighter-rouge">ELProcessor</code>, installing a custom resolver is only a few lines of code. Writing the resolver carefully is the important part:</p>

<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">public</span> <span class="kd">final</span> <span class="kd">class</span> <span class="nc">SafeELEvaluator</span> <span class="o">{</span>

    <span class="kd">private</span> <span class="kd">final</span> <span class="nc">ELProcessor</span> <span class="n">template</span><span class="o">;</span>   <span class="c1">// pre-built once; reused per call</span>

    <span class="kd">public</span> <span class="nf">SafeELEvaluator</span><span class="o">()</span> <span class="o">{</span>
        <span class="k">this</span><span class="o">.</span><span class="na">template</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">ELProcessor</span><span class="o">();</span>
        <span class="k">this</span><span class="o">.</span><span class="na">template</span><span class="o">.</span><span class="na">getELManager</span><span class="o">().</span><span class="na">addELResolver</span><span class="o">(</span><span class="k">new</span> <span class="nc">SafeELResolver</span><span class="o">());</span>
        <span class="c1">// SafeELResolver:</span>
        <span class="c1">//   - whitelists base types (Map, Collection, CharSequence, Number,</span>
        <span class="c1">//     Boolean, the java.time types, and the `fn` helper); domain</span>
        <span class="c1">//     records are flattened to maps, so EL never touches domain methods</span>
        <span class="c1">//   - blocks .class / getClass / forName / Runtime / System / Thread / exit</span>
        <span class="c1">//   - blocks method invocation through the resolver where needed</span>
        <span class="c1">//   - caps the number of resolutions per evaluation (a DoS budget)</span>
    <span class="o">}</span>

    <span class="kd">public</span> <span class="kt">boolean</span> <span class="nf">evaluate</span><span class="o">(</span><span class="nc">String</span> <span class="n">predicate</span><span class="o">,</span> <span class="nc">Speaker</span> <span class="n">speaker</span><span class="o">,</span> <span class="nc">ConfEvent</span> <span class="n">event</span><span class="o">)</span> <span class="o">{</span>
        <span class="nc">ELProcessor</span> <span class="n">el</span> <span class="o">=</span> <span class="n">cloneTemplate</span><span class="o">();</span>          <span class="c1">// cheap; per-call processor</span>
        <span class="n">el</span><span class="o">.</span><span class="na">defineBean</span><span class="o">(</span><span class="s">"speaker"</span><span class="o">,</span> <span class="n">toMap</span><span class="o">(</span><span class="n">speaker</span><span class="o">));</span>  <span class="c1">// per-call map; no domain methods exposed</span>
        <span class="n">el</span><span class="o">.</span><span class="na">defineBean</span><span class="o">(</span><span class="s">"event"</span><span class="o">,</span>   <span class="n">toMap</span><span class="o">(</span><span class="n">event</span><span class="o">));</span>
        <span class="k">return</span> <span class="nf">runWithTimeout</span><span class="o">(</span><span class="mi">50</span><span class="o">,</span> <span class="no">MILLISECONDS</span><span class="o">,</span>
                <span class="o">()</span> <span class="o">-&gt;</span> <span class="nc">Boolean</span><span class="o">.</span><span class="na">TRUE</span><span class="o">.</span><span class="na">equals</span><span class="o">(</span><span class="n">el</span><span class="o">.</span><span class="na">eval</span><span class="o">(</span><span class="n">predicate</span><span class="o">)));</span>
    <span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>

<p>Three things are doing the work here:</p>

<ol>
  <li><strong><code class="language-plaintext highlighter-rouge">defineBean(...)</code></strong> binds <em>just</em> the roots the policy can see &#8211; and binds <code class="language-plaintext highlighter-rouge">speaker</code> and <code class="language-plaintext highlighter-rouge">event</code> as small per-call maps, so there are no domain record methods left to navigate. Nothing else from the application is reachable.</li>
  <li><strong>A custom <code class="language-plaintext highlighter-rouge">ELResolver</code></strong>, registered before the defaults, <strong>rejects</strong> any property lookup that would reach <code class="language-plaintext highlighter-rouge">getClass</code>, <code class="language-plaintext highlighter-rouge">java.lang.*</code>, or types that are not on the whitelist. Writing a strict-enough <code class="language-plaintext highlighter-rouge">ELResolver</code> is the part that is easy to under-think. Whitelist <em>types</em> (not packages), block method invocation through the resolver where needed, refuse anything that smells like <code class="language-plaintext highlighter-rouge">class</code>, <code class="language-plaintext highlighter-rouge">getClass</code>, or <code class="language-plaintext highlighter-rouge">forName</code>. When in doubt, deny.</li>
  <li><strong>A wall-clock timeout</strong> caps how long a single predicate can run. A predicate that triggers expensive resolution, deep nesting, or recursive helper calls is a denial-of-service vector even when it can&#8217;t escape the sandbox.</li>
</ol>

<p><em><strong>Note:</strong> the <code class="language-plaintext highlighter-rouge">SafeELEvaluator</code> above is simplified for the post &#8211; the <a href="https://github.com/mehmandarov/jakarta-hidden-gems/blob/main/03-el-policy/src/main/java/com/mehmandarov/jhg/el/SafeELEvaluator.java">shipped class</a> builds a fresh <code class="language-plaintext highlighter-rouge">ELProcessor</code> per call, installs a new <a href="https://github.com/mehmandarov/jakarta-hidden-gems/blob/main/03-el-policy/src/main/java/com/mehmandarov/jhg/el/SafeELResolver.java"><code class="language-plaintext highlighter-rouge">SafeELResolver</code></a> at the front each time, imports the allowed JDK time types, and runs every evaluation through the <a href="https://github.com/mehmandarov/jakarta-hidden-gems/blob/main/03-el-policy/src/main/java/com/mehmandarov/jhg/el/Timeouts.java"><code class="language-plaintext highlighter-rouge">Timeouts</code></a> helper. See the <a href="https://github.com/mehmandarov/jakarta-hidden-gems/blob/main/03-el-policy/src/main/java/com/mehmandarov/jhg/el/SafeELEvaluator.java">real code</a> for the full picture.</em></p>

<h3 id="23-replacing-the-parserevaluator-with-jakarta-el">2.3 Replacing the parser/evaluator with Jakarta EL</h3>

<p>Gem #2&#8217;s <code class="language-plaintext highlighter-rouge">PredicateEngine</code> is a concrete CDI bean (it delegates to the handwritten <code class="language-plaintext highlighter-rouge">PredicateEvaluator</code>), and <code class="language-plaintext highlighter-rouge">ElPredicateEngine</code> plugs in ahead of it via a globally-enabled CDI <code class="language-plaintext highlighter-rouge">@Alternative</code> + <code class="language-plaintext highlighter-rouge">@Priority</code> &#8211; the portable, <em>Quarkus-friendly</em> equivalent of <code class="language-plaintext highlighter-rouge">@Specializes</code>. Quarkus&#8217;s CDI engine, Arc, does not support <code class="language-plaintext highlighter-rouge">@Specializes</code>, so the <code class="language-plaintext highlighter-rouge">@Alternative</code> route is what keeps the same gem source running unchanged on Liberty, Helidon <strong>and</strong> Quarkus.</p>

<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nd">@Alternative</span>
<span class="nd">@Priority</span><span class="o">(</span><span class="mi">1</span><span class="o">)</span>
<span class="nd">@ApplicationScoped</span>
<span class="kd">public</span> <span class="kd">class</span> <span class="nc">ElPredicateEngine</span> <span class="kd">extends</span> <span class="nc">PredicateEngine</span> <span class="o">{</span>
    <span class="nd">@Override</span>
    <span class="kd">public</span> <span class="kt">boolean</span> <span class="nf">evaluate</span><span class="o">(</span><span class="nc">String</span> <span class="n">predicate</span><span class="o">,</span> <span class="nc">Speaker</span> <span class="n">speaker</span><span class="o">,</span> <span class="nc">ConfEvent</span> <span class="n">event</span><span class="o">)</span> <span class="o">{</span>
        <span class="k">if</span> <span class="o">(</span><span class="n">predicate</span> <span class="o">==</span> <span class="kc">null</span> <span class="o">||</span> <span class="n">predicate</span><span class="o">.</span><span class="na">isBlank</span><span class="o">())</span> <span class="k">return</span> <span class="kc">true</span><span class="o">;</span>   <span class="c1">// empty == allow</span>
        <span class="nc">String</span> <span class="n">el</span> <span class="o">=</span> <span class="n">toEl</span><span class="o">(</span><span class="n">predicate</span><span class="o">);</span>                  <span class="c1">// rewrite gem #2's infix operators</span>
        <span class="k">if</span> <span class="o">(</span><span class="no">BLOCKED</span><span class="o">.</span><span class="na">matcher</span><span class="o">(</span><span class="n">el</span><span class="o">).</span><span class="na">find</span><span class="o">())</span> <span class="k">return</span> <span class="kc">false</span><span class="o">;</span> <span class="c1">// fail-closed denylist</span>
        <span class="k">try</span> <span class="o">{</span>
            <span class="nc">ELProcessor</span> <span class="n">p</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">ELProcessor</span><span class="o">();</span>
            <span class="n">p</span><span class="o">.</span><span class="na">defineBean</span><span class="o">(</span><span class="s">"speaker"</span><span class="o">,</span> <span class="n">toMap</span><span class="o">(</span><span class="n">speaker</span><span class="o">));</span>  <span class="c1">// per-call maps; no domain methods exposed</span>
            <span class="n">p</span><span class="o">.</span><span class="na">defineBean</span><span class="o">(</span><span class="s">"event"</span><span class="o">,</span>   <span class="n">toMap</span><span class="o">(</span><span class="n">event</span><span class="o">));</span>
            <span class="n">p</span><span class="o">.</span><span class="na">defineBean</span><span class="o">(</span><span class="s">"fn"</span><span class="o">,</span>      <span class="no">FUNCTIONS</span><span class="o">);</span>       <span class="c1">// helper functions</span>
            <span class="k">return</span> <span class="nc">Boolean</span><span class="o">.</span><span class="na">TRUE</span><span class="o">.</span><span class="na">equals</span><span class="o">(</span><span class="n">p</span><span class="o">.</span><span class="na">eval</span><span class="o">(</span><span class="n">el</span><span class="o">));</span>
        <span class="o">}</span> <span class="k">catch</span> <span class="o">(</span><span class="nc">RuntimeException</span> <span class="n">e</span><span class="o">)</span> <span class="o">{</span>
            <span class="k">return</span> <span class="kc">false</span><span class="o">;</span>                             <span class="c1">// malformed/blocked predicate &#8594; deny</span>
        <span class="o">}</span>
    <span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>

<p>The handwritten <code class="language-plaintext highlighter-rouge">PredicateEvaluator</code> from gem #2 stays exactly as it shipped &#8211; the <code class="language-plaintext highlighter-rouge">@Alternative</code> swaps the <em>engine</em> behind the same <code class="language-plaintext highlighter-rouge">AccessPolicy</code>, so nothing downstream changes. Old rules keep working because <code class="language-plaintext highlighter-rouge">toEl(...)</code> rewrites gem #2&#8217;s infix <code class="language-plaintext highlighter-rouge">intersects</code>/<code class="language-plaintext highlighter-rouge">contains</code> operators into <code class="language-plaintext highlighter-rouge">fn.*</code> calls before evaluation &#8211; Jakarta EL has no infix <code class="language-plaintext highlighter-rouge">intersects</code>, so it&#8217;s that small translation step, not native parsing, that preserves backward compatibility.</p>

<hr />

<h2 id="3-trade-offs-and-when-to-use-what">3. Trade-offs and when to use what</h2>

<h3 id="31-what-ships-today-vs-where-were-headed">3.1 What ships today vs. where we&#8217;re headed</h3>

<p>Gem #3 in <a href="https://github.com/mehmandarov/jakarta-hidden-gems">the repo</a> ships <em>both</em> engines. The running default, <a href="https://github.com/mehmandarov/jakarta-hidden-gems/blob/main/03-el-policy/src/main/java/com/mehmandarov/jhg/el/ElPredicateEngine.java"><code class="language-plaintext highlighter-rouge">ElPredicateEngine</code></a> from above, exposes <code class="language-plaintext highlighter-rouge">speaker</code> and <code class="language-plaintext highlighter-rouge">event</code> as small per-call maps (so policy expressions do not navigate the domain records directly), runs a fail-closed <em>denylist</em> over the expression, and adds a small <code class="language-plaintext highlighter-rouge">fn</code> helper bean (<code class="language-plaintext highlighter-rouge">fn.daysUntil(...)</code>, <code class="language-plaintext highlighter-rouge">fn.intersects(...)</code>). That is the pragmatic baseline you can run right now.</p>

<p>Shipping right next to it is the <em>hardened</em> <a href="https://github.com/mehmandarov/jakarta-hidden-gems/blob/main/03-el-policy/src/main/java/com/mehmandarov/jhg/el/SafeELEvaluator.java"><code class="language-plaintext highlighter-rouge">SafeELEvaluator</code></a> from above &#8211; a real <a href="https://github.com/mehmandarov/jakarta-hidden-gems/blob/main/03-el-policy/src/main/java/com/mehmandarov/jhg/el/SafeELResolver.java"><code class="language-plaintext highlighter-rouge">SafeELResolver</code></a> that whitelists <em>types</em> instead of denylisting strings, plus a per-call wall-clock timeout (the <a href="https://github.com/mehmandarov/jakarta-hidden-gems/blob/main/03-el-policy/src/main/java/com/mehmandarov/jhg/el/Timeouts.java"><code class="language-plaintext highlighter-rouge">Timeouts</code></a> helper, since EL has no execution budget of its own). The denylist gets you started; the resolver is where you want to be.</p>

<h3 id="32-is-the-swap-worth-it">3.2 Is the swap worth it?</h3>

<p>So, is swapping the handwritten parser/evaluator for a sandboxed <code class="language-plaintext highlighter-rouge">ELProcessor</code> worth it? Here&#8217;s how the trade-offs net out:</p>

<p><strong>&#9989; Pros:</strong></p>

<ul>
  <li><strong>No parser to maintain.</strong> <code class="language-plaintext highlighter-rouge">ELProcessor</code> ships with the platform; you delete the hand-rolled tokeniser, operator enum, and reflection helper.</li>
  <li><strong>Comparable size for the baseline, and the rules carry over.</strong> The denylist <code class="language-plaintext highlighter-rouge">ElPredicateEngine</code> (~110 lines) plus its <code class="language-plaintext highlighter-rouge">fn</code> helpers lands in the same ballpark as the ~160-line parser it replaces, and gem #2&#8217;s existing rules keep working unchanged.</li>
  <li><strong>Fits hot-reloadable policy files.</strong> EL is a string; the policy file is a string; pair them with a <code class="language-plaintext highlighter-rouge">WatchService</code> and you have live policy edits.</li>
  <li><strong>Richer policies without redeploys.</strong> Date arithmetic, member access, even ternaries are all available.</li>
  <li><strong>The sandbox is local and auditable.</strong> Your <code class="language-plaintext highlighter-rouge">SafeELResolver</code> is the only thing that decides what&#8217;s reachable, and you can read it on one screen.</li>
</ul>

<p><strong>&#10060; Cons:</strong></p>

<ul>
  <li>You&#8217;re now responsible for the resolver whitelist. Get it wrong and you are back in <a href="#21-how-dangerous-can-it-be-why-bother-validating-the-input">unvalidated-input territory</a>.</li>
  <li><strong>The hardened path is more code, not less.</strong> The whitelist <code class="language-plaintext highlighter-rouge">SafeELResolver</code> (~190 lines) plus the <code class="language-plaintext highlighter-rouge">Timeouts</code> watchdog is larger than the parser &#8211; you&#8217;re trading raw line count for a single, auditable security surface.</li>
  <li>A malformed predicate still fails <em>closed</em> &#8211; the engine wraps <code class="language-plaintext highlighter-rouge">eval(...)</code> in a <code class="language-plaintext highlighter-rouge">try/catch</code> that denies and logs a warning &#8211; but the logged reason reads like an EL error, not your own. Map <code class="language-plaintext highlighter-rouge">ELException</code> to a friendlier &#8220;rule X failed to evaluate&#8221; if your operators care.</li>
  <li>A <a href="https://github.com/mehmandarov/jakarta-hidden-gems/blob/main/03-el-policy/src/main/java/com/mehmandarov/jhg/el/Timeouts.java">timeout watchdog</a> is <strong>not optional</strong>. EL has no built-in execution budget.</li>
</ul>

<h3 id="33-summary-comparison">3.3 Summary comparison</h3>

<table class="bordered-table">
  <thead>
    <tr>
      <th>Option</th>
      <th>What it gives you</th>
      <th>Surface to audit</th>
      <th>When to reach for it</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td><strong>Handwritten <code class="language-plaintext highlighter-rouge">PredicateEvaluator</code></strong> <em>(gem #2)</em></td>
      <td>A strict four-operator subset. No nesting, no method calls.</td>
      <td>~160 lines of parser + reflection.</td>
      <td>When you want a tiny evaluator surface and accept the feature ceiling.</td>
    </tr>
    <tr>
      <td><strong><code class="language-plaintext highlighter-rouge">ElPredicateEngine</code></strong> <em>(gem #3 default)</em></td>
      <td>EL-backed predicates, old <code class="language-plaintext highlighter-rouge">intersects</code>/<code class="language-plaintext highlighter-rouge">contains</code> rules rewritten to <code class="language-plaintext highlighter-rouge">fn.*</code>, plus a fail-closed denylist.</td>
      <td>~110-line engine + helper functions + denylist.</td>
      <td>When you want the migration path and backwards compatibility with gem #2 rules.</td>
    </tr>
    <tr>
      <td><strong><code class="language-plaintext highlighter-rouge">SafeELEvaluator</code></strong> <em>(hardened path)</em></td>
      <td>Full EL semantics, restricted to whitelisted types and bound beans, with a per-call timeout.</td>
      <td>~190-line <code class="language-plaintext highlighter-rouge">ELResolver</code> + a watchdog (<code class="language-plaintext highlighter-rouge">Timeouts</code>).</td>
      <td>When policy authors need more than four operators and you can own the resolver.</td>
    </tr>
    <tr>
      <td><strong>Raw <code class="language-plaintext highlighter-rouge">ELProcessor</code></strong></td>
      <td>Full EL, default resolver chain, no timeout.</td>
      <td>Default resolver chain + everything reachable from exposed objects.</td>
      <td><strong>Don&#8217;t</strong>, unless every input is application-controlled.</td>
    </tr>
    <tr>
      <td><strong>Drools / OPA / Cedar</strong></td>
      <td>A real rules engine with its own language, debugger, and tooling.</td>
      <td>An external service or a multi-MB dependency.</td>
      <td>When your rules language is the product, not a side quest.</td>
    </tr>
  </tbody>
</table>

<hr />

<h2 id="conclusion">Conclusion</h2>

<p>There is no silver bullet for &#8220;embed a small rules language in a Jakarta app&#8221;, but there is a workable progression:</p>

<ul>
  <li>For <strong>a handful of trivial predicates</strong>, the handwritten approach from gem #2 is honest about its limits and has a tiny evaluator surface. Ship it, move on.</li>
  <li>For <strong>anything that wants method calls, date arithmetic, or member access</strong>, an EL-backed evaluator is, in my opinion, a good trade. The pragmatic <code class="language-plaintext highlighter-rouge">ElPredicateEngine</code> replaces the parser with a similarly sized, more capable engine while keeping the policy file format the same. The hardened path may not reduce line count, but it moves the risk into one resolver and one timeout boundary &#8211; a smaller and more auditable security surface.</li>
  <li>For <strong>a rules language that is the product</strong> (think compliance engines, fraud rules, complex insurance policies), reach for Drools or OPA. EL is not trying to be that.</li>
</ul>

<p>You may have heard about CVEs related to Jakarta EL, and those issues were real. The pattern behind the named issues was <em>unvalidated input</em> reaching an evaluator with too much access, and they were patched years ago in supported versions. &#8220;Don&#8217;t feed user input to a <code class="language-plaintext highlighter-rouge">${...}</code>&#8221; is a rule that bears repeating &#8211; but it&#8217;s the same rule as for SQL or the shell, not something peculiar to EL. Once you own the resolver chain and only ever evaluate trusted policy strings, EL stops being a liability and turns into a fairly under-used corner of the platform.</p>

<p>I picked the sandboxed-EL approach for the demo because it does what the rest of the series cares about: <strong>less custom code, more standard platform, no runtime-specific extension</strong>. Gem #3 ships it as <code class="language-plaintext highlighter-rouge">ElPredicateEngine</code> and it hardens into <code class="language-plaintext highlighter-rouge">SafeELEvaluator</code> &#8211; and the same engine runs unchanged on Quarkus, Helidon, and Open Liberty.</p>

<h2 id="whats-next">What&#8217;s Next?</h2>

<p>If REST API design is your thing, my posts on <a href="/api-versioning/">API versioning</a> and <a href="/rfc-9457-problem-details-jakarta-ee/">RFC 9457 Problem Details</a> are also worth a look. The full code for this post &#8211; both engines, the resolver, and the watchdog &#8211; lives in gem #3 of the <a href="https://github.com/mehmandarov/jakarta-hidden-gems">Jakarta EE Hidden Gems</a> repository. Also, keep an eye on the <a href="/tag/jakarta-ee/">Jakarta EE tag</a> here in general.</p>

<p><strong><em>Happy (and safe) evaluating, folks!</em></strong></p>

<hr />]]></content><author><name>Rustam Mehmandarov</name></author><summary type="html">A practical look at Jakarta Expression Language as a small in-process policy engine &#8211; replacing a handwritten predicate parser with `jakarta.el.ELProcessor`, then hardening evaluation with a resolver whitelist and timeout.</summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://mehmandarov.com/assets/images/posts-images/books.jpg"/><category term="blog"/><category term="english"/><category term="java"/><category term="jakarta ee"/><category term="security"/><category term="architecture"/></entry><entry><title type="html">Sane API error handling with RFC 9457 Problem Details in Jakarta EE</title><link href="https://mehmandarov.com/rfc-9457-problem-details-jakarta-ee/" rel="alternate" type="text/html" title="Sane API error handling with RFC 9457 Problem Details in Jakarta EE"/><published>2026-05-25T10:00:00+02:00</published><updated>2026-05-25T10:00:00+02:00</updated><id>https://mehmandarov.com/rfc-9457-problem-details-jakarta-ee</id><content type="html" xml:base="https://mehmandarov.com/rfc-9457-problem-details-jakarta-ee/"><![CDATA[<p><em>When APIs end up with their own error format, it quickly gets annoying for anyone who has to consume more than one API. <a href="https://www.rfc-editor.org/rfc/rfc9457">RFC 9457</a> defines a standard envelope for HTTP API errors. Let&#8217;s have a look at how to do it in Jakarta EE: a small hand-made <code class="language-plaintext highlighter-rouge">ProblemDetail</code> plus one <code class="language-plaintext highlighter-rouge">ExceptionMapper</code> per error category; with the <a href="https://github.com/zalando/problem">Zalando Problem</a> library; followed by quick notes on Quarkus and Spring as alternatives.</em></p>

<ul>
  <li><a href="#introduction">Introduction</a></li>
  <li><a href="#tldr-why-rfc-9457">TL;DR: Why RFC 9457?</a></li>
  <li><a href="#lets-write-some-code">Let&#8217;s write some code!</a>
    <ul>
      <li><a href="#1-hand-made-problemdetail--exceptionmapper">1. Hand-made <code class="language-plaintext highlighter-rouge">ProblemDetail</code> + <code class="language-plaintext highlighter-rouge">ExceptionMapper</code></a></li>
      <li><a href="#2-zalando-problem">2. Zalando Problem</a></li>
      <li><a href="#3-quarkus-quarkus-http-problem">3. Quarkus: <code class="language-plaintext highlighter-rouge">quarkus-http-problem</code></a></li>
      <li><a href="#4-spring-boot--a-short-note">4. Spring Boot &#8211; a short note</a></li>
    </ul>
  </li>
  <li><a href="#conclusion">Conclusion</a></li>
  <li><a href="#whats-next">What&#8217;s Next?</a></li>
</ul>

<hr />

<h2 id="introduction">Introduction</h2>

<p>If you&#8217;ve consumed more than one or two REST APIs, you&#8217;ve seen the pattern. One service returns <code class="language-plaintext highlighter-rouge">{"error": "..."}</code>, another <code class="language-plaintext highlighter-rouge">{"message": "...", "code": 42}</code>, a third returns <code class="language-plaintext highlighter-rouge">200 OK</code> with an error hidden somewhere deep in the response. Your REST client code fills up with special cases for each one. Sounds familiar?</p>

<p><a href="https://www.rfc-editor.org/rfc/rfc9457">RFC 9457 &#8211; Problem Details for HTTP APIs</a> (the successor to RFC 7807) defines a single JSON envelope for errors, served as <code class="language-plaintext highlighter-rouge">application/problem+json</code> MIME type. It is a small spec: five well-defined bits of information and an <code class="language-plaintext highlighter-rouge">extensions</code> map for anything else you might need.</p>

<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w">
  </span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"urn:problem-type:validation-error"</span><span class="p">,</span><span class="w">
  </span><span class="nl">"title"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Validation Failed"</span><span class="p">,</span><span class="w">
  </span><span class="nl">"status"</span><span class="p">:</span><span class="w"> </span><span class="mi">400</span><span class="p">,</span><span class="w">
  </span><span class="nl">"detail"</span><span class="p">:</span><span class="w"> </span><span class="s2">"The request body or parameters failed validation."</span><span class="p">,</span><span class="w">
  </span><span class="nl">"extensions"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
    </span><span class="nl">"violations"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
      </span><span class="p">{</span><span class="w"> </span><span class="nl">"field"</span><span class="p">:</span><span class="w"> </span><span class="s2">"title"</span><span class="p">,</span><span class="w"> </span><span class="nl">"message"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Title is required"</span><span class="w"> </span><span class="p">}</span><span class="w">
    </span><span class="p">]</span><span class="w">
  </span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>

<h2 id="tldr-why-rfc-9457">TL;DR: Why RFC 9457?</h2>

<p>Why not keep creating your own?</p>

<ul>
  <li><strong>Consumers already know the shape.</strong> Generated SDKs, gateways, log pipelines, and tracing tools can parse <code class="language-plaintext highlighter-rouge">application/problem+json</code> without extra work.</li>
  <li><strong>You can extend it without breaking clients.</strong> The <code class="language-plaintext highlighter-rouge">extensions</code> map is part of the spec &#8211; put what you need in there.</li>
  <li><strong>It separates the <em>category</em> from the <em>instance</em>.</strong> <code class="language-plaintext highlighter-rouge">type</code> says &#8220;this is a validation error&#8221; (stable, machine-readable); <code class="language-plaintext highlighter-rouge">detail</code> and <code class="language-plaintext highlighter-rouge">instance</code> describe what happened <em>this</em> time.</li>
</ul>

<p>&#128161; <em><strong>Note:</strong> RFC 9457 is just a JSON structure and a content type. No library or framework is required. That&#8217;s why there are so many implementations &#8211; and why a hand-made one is often a reasonable choice.</em></p>

<hr />

<h2 id="lets-write-some-code">Let&#8217;s write some code!</h2>

<p>I have created a repository called <a href="https://github.com/mehmandarov/api-guide-java">API Guide for Java</a> to showcase the patterns for one of my talks. For this post, have a look at <a href="https://github.com/mehmandarov/api-guide-java/blob/main/src/main/java/com/mehmandarov/confapi/error/ProblemDetail.java"><code class="language-plaintext highlighter-rouge">ProblemDetail.java</code></a> and the mappers next to it under <a href="https://github.com/mehmandarov/api-guide-java/tree/main/src/main/java/com/mehmandarov/confapi/error"><code class="language-plaintext highlighter-rouge">com/mehmandarov/confapi/error/</code></a>.</p>

<h3 id="1-hand-made-problemdetail--exceptionmapper">1. Hand-made <code class="language-plaintext highlighter-rouge">ProblemDetail</code> + <code class="language-plaintext highlighter-rouge">ExceptionMapper</code></h3>

<h4 id="what-it-looks-like">What it looks like</h4>

<p>Imagine you have a REST interface looking like this:</p>

<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nd">@GET</span>
<span class="nd">@Path</span><span class="o">(</span><span class="s">"/{id}"</span><span class="o">)</span>
<span class="nd">@Operation</span><span class="o">(</span><span class="n">summary</span> <span class="o">=</span> <span class="s">"Get room by ID"</span><span class="o">)</span>
<span class="nd">@APIResponse</span><span class="o">(</span><span class="n">responseCode</span> <span class="o">=</span> <span class="s">"200"</span><span class="o">,</span> <span class="n">description</span> <span class="o">=</span> <span class="s">"Room found"</span><span class="o">)</span>
<span class="nd">@APIResponse</span><span class="o">(</span><span class="n">responseCode</span> <span class="o">=</span> <span class="s">"404"</span><span class="o">,</span> <span class="n">description</span> <span class="o">=</span> <span class="s">"Room not found"</span><span class="o">)</span>
<span class="kd">public</span> <span class="nc">Room</span> <span class="nf">getById</span><span class="o">(</span>
        <span class="nd">@Parameter</span><span class="o">(</span><span class="n">description</span> <span class="o">=</span> <span class="s">"Room ID"</span><span class="o">,</span> <span class="n">required</span> <span class="o">=</span> <span class="kc">true</span><span class="o">)</span>
        <span class="nd">@PathParam</span><span class="o">(</span><span class="s">"id"</span><span class="o">)</span> <span class="nc">String</span> <span class="n">id</span><span class="o">)</span> <span class="o">{</span>
    <span class="k">return</span> <span class="n">repo</span><span class="o">.</span><span class="na">findById</span><span class="o">(</span><span class="n">id</span><span class="o">)</span>
            <span class="o">.</span><span class="na">orElseThrow</span><span class="o">(()</span> <span class="o">-&gt;</span> <span class="k">new</span> <span class="nc">NotFoundException</span><span class="o">(</span><span class="s">"Room not found: "</span> <span class="o">+</span> <span class="n">id</span><span class="o">));</span>
<span class="o">}</span>
</code></pre></div></div>

<p>Now, you can add a single <code class="language-plaintext highlighter-rouge">ProblemDetail</code> class &#8211; built around the five RFC 9457 elements and an <code class="language-plaintext highlighter-rouge">extensions</code> map &#8211; and one <a href="https://jakarta.ee/specifications/restful-ws/4.0/apidocs/jakarta.ws.rs/jakarta/ws/rs/ext/exceptionmapper"><code class="language-plaintext highlighter-rouge">ExceptionMapper</code></a> per error category.</p>

<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">public</span> <span class="kd">class</span> <span class="nc">ProblemDetail</span> <span class="o">{</span>
    <span class="kd">private</span> <span class="no">URI</span> <span class="n">type</span> <span class="o">=</span> <span class="no">URI</span><span class="o">.</span><span class="na">create</span><span class="o">(</span><span class="s">"about:blank"</span><span class="o">);</span>
    <span class="kd">private</span> <span class="nc">String</span> <span class="n">title</span><span class="o">;</span>
    <span class="kd">private</span> <span class="kt">int</span> <span class="n">status</span><span class="o">;</span>
    <span class="kd">private</span> <span class="nc">String</span> <span class="n">detail</span><span class="o">;</span>
    <span class="kd">private</span> <span class="no">URI</span> <span class="n">instance</span><span class="o">;</span>
    <span class="kd">private</span> <span class="kd">final</span> <span class="nc">Map</span><span class="o">&lt;</span><span class="nc">String</span><span class="o">,</span> <span class="nc">Object</span><span class="o">&gt;</span> <span class="n">extensions</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">LinkedHashMap</span><span class="o">&lt;&gt;();</span>

    <span class="kd">public</span> <span class="kd">static</span> <span class="nc">ProblemDetail</span> <span class="nf">of</span><span class="o">(</span><span class="kt">int</span> <span class="n">status</span><span class="o">,</span> <span class="nc">String</span> <span class="n">title</span><span class="o">)</span> <span class="o">{</span> <span class="cm">/* ... */</span> <span class="o">}</span>
    <span class="kd">public</span> <span class="nc">ProblemDetail</span> <span class="nf">withType</span><span class="o">(</span><span class="nc">String</span> <span class="n">typeUri</span><span class="o">)</span>            <span class="o">{</span> <span class="cm">/* ... */</span> <span class="o">}</span>
    <span class="kd">public</span> <span class="nc">ProblemDetail</span> <span class="nf">withExtension</span><span class="o">(</span><span class="nc">String</span> <span class="n">key</span><span class="o">,</span> <span class="nc">Object</span> <span class="n">v</span><span class="o">)</span> <span class="o">{</span> <span class="cm">/* ... */</span> <span class="o">}</span>
    <span class="c1">// + getters/setters</span>
<span class="o">}</span>
</code></pre></div></div>

<p>The interesting part is how it gets used. As you can see from the resource code above, there is <strong>no <code class="language-plaintext highlighter-rouge">try/catch</code> in resources, ever</strong> &#8211; every exception is turned into a Problem Details response by an <code class="language-plaintext highlighter-rouge">ExceptionMapper</code>:</p>

<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nd">@Provider</span>
<span class="kd">public</span> <span class="kd">class</span> <span class="nc">ConstraintViolationExceptionMapper</span>
        <span class="kd">implements</span> <span class="nc">ExceptionMapper</span><span class="o">&lt;</span><span class="nc">ConstraintViolationException</span><span class="o">&gt;</span> <span class="o">{</span>

    <span class="nd">@Override</span>
    <span class="kd">public</span> <span class="nc">Response</span> <span class="nf">toResponse</span><span class="o">(</span><span class="nc">ConstraintViolationException</span> <span class="n">ex</span><span class="o">)</span> <span class="o">{</span>
        <span class="nc">List</span><span class="o">&lt;</span><span class="nc">Map</span><span class="o">&lt;</span><span class="nc">String</span><span class="o">,</span> <span class="nc">String</span><span class="o">&gt;&gt;</span> <span class="n">violations</span> <span class="o">=</span> <span class="n">ex</span><span class="o">.</span><span class="na">getConstraintViolations</span><span class="o">()</span>
                <span class="o">.</span><span class="na">stream</span><span class="o">().</span><span class="na">map</span><span class="o">(</span><span class="k">this</span><span class="o">::</span><span class="n">toMap</span><span class="o">).</span><span class="na">toList</span><span class="o">();</span>

        <span class="nc">ProblemDetail</span> <span class="n">problem</span> <span class="o">=</span> <span class="nc">ProblemDetail</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="mi">400</span><span class="o">,</span> <span class="s">"Validation Failed"</span><span class="o">)</span>
            <span class="o">.</span><span class="na">withType</span><span class="o">(</span><span class="s">"urn:problem-type:validation-error"</span><span class="o">)</span>
            <span class="o">.</span><span class="na">withExtension</span><span class="o">(</span><span class="s">"violations"</span><span class="o">,</span> <span class="n">violations</span><span class="o">);</span>

        <span class="k">return</span> <span class="nc">Response</span><span class="o">.</span><span class="na">status</span><span class="o">(</span><span class="mi">400</span><span class="o">)</span>
                <span class="o">.</span><span class="na">type</span><span class="o">(</span><span class="s">"application/problem+json"</span><span class="o">)</span>
                <span class="o">.</span><span class="na">entity</span><span class="o">(</span><span class="n">problem</span><span class="o">).</span><span class="na">build</span><span class="o">();</span>
    <span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>

<p>One mapper per category keeps each file small and obvious: <code class="language-plaintext highlighter-rouge">ConstraintViolationExceptionMapper</code> &#8594; 400, <code class="language-plaintext highlighter-rouge">NotFoundExceptionMapper</code> &#8594; 404, <code class="language-plaintext highlighter-rouge">NotAuthorizedExceptionMapper</code> &#8594; 401, <code class="language-plaintext highlighter-rouge">ForbiddenExceptionMapper</code> &#8594; 403, and a <code class="language-plaintext highlighter-rouge">CatchAllExceptionMapper</code> &#8594; 500 that <strong>never leaks stack traces</strong> to clients.</p>

<p>&#9888;&#65039;<em><strong>A word of caution:</strong> The catch-all mapper is the safety net for everything you forgot to handle. Without one, an uncaught exception ends up in the server&#8217;s default error page, which often includes stack traces, server versions, and sometimes filesystem paths. However, it might be a good idea to handle most of the common exceptions explicitly, and leave the generic catch-all for something truly unexpected.</em></p>

<p><strong>&#9989; Pros:</strong></p>

<ul>
  <li><strong>Portable across runtimes.</strong> The same code runs on Quarkus, Helidon, and Open Liberty. No runtime-specific extension.</li>
  <li><strong>No extra dependencies.</strong> RFC 9457 is just a JSON structure; you don&#8217;t need a library to emit one.</li>
  <li><strong>Small, readable surface.</strong> The error model fits on one slide. When something goes wrong, you can read the source.</li>
</ul>

<p><strong>&#10060; Cons:</strong></p>

<ul>
  <li>You write the boilerplate yourself &#8211; one mapper per category.</li>
  <li>Nothing maps validation, <code class="language-plaintext highlighter-rouge">WebApplicationException</code>, or uncaught <code class="language-plaintext highlighter-rouge">Throwable</code> automatically &#8211; you wire each one up. (This can also be one of the pros, depending on the way you look at things.)</li>
  <li>No content negotiation between <code class="language-plaintext highlighter-rouge">application/json</code> and <code class="language-plaintext highlighter-rouge">application/problem+json</code> unless you add it yourself. (Spring, for example, has a built-in <code class="language-plaintext highlighter-rouge">ProblemDetail</code> that does this for you.)</li>
</ul>

<p>&#128161; <em><strong>Want to know more?</strong> The full code, including all five mappers, lives in <a href="https://github.com/mehmandarov/api-guide-java/tree/main/src/main/java/com/mehmandarov/confapi/error"><code class="language-plaintext highlighter-rouge">com/mehmandarov/confapi/error/</code></a>.</em></p>

<hr />

<h3 id="2-zalando-problem">2. Zalando Problem</h3>

<h4 id="what-it-looks-like-1">What it looks like</h4>

<p>The <a href="https://github.com/zalando/problem">Zalando Problem</a> library (<code class="language-plaintext highlighter-rouge">org.zalando:problem</code> + <code class="language-plaintext highlighter-rouge">jackson-datatype-problem</code>) gives you <code class="language-plaintext highlighter-rouge">Problem</code> and <code class="language-plaintext highlighter-rouge">ThrowableProblem</code> types and Jackson serialization. You still write an <code class="language-plaintext highlighter-rouge">ExceptionMapper</code> to bridge JAX-RS exceptions to <code class="language-plaintext highlighter-rouge">Problem</code>, but you don&#8217;t define the envelope yourself.</p>

<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">org.zalando.problem.Problem</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">org.zalando.problem.Status</span><span class="o">;</span>

<span class="nc">Problem</span> <span class="n">problem</span> <span class="o">=</span> <span class="nc">Problem</span><span class="o">.</span><span class="na">builder</span><span class="o">()</span>
        <span class="o">.</span><span class="na">withType</span><span class="o">(</span><span class="no">URI</span><span class="o">.</span><span class="na">create</span><span class="o">(</span><span class="s">"urn:problem-type:validation-error"</span><span class="o">))</span>
        <span class="o">.</span><span class="na">withTitle</span><span class="o">(</span><span class="s">"Validation Failed"</span><span class="o">)</span>
        <span class="o">.</span><span class="na">withStatus</span><span class="o">(</span><span class="nc">Status</span><span class="o">.</span><span class="na">BAD_REQUEST</span><span class="o">)</span>
        <span class="o">.</span><span class="na">with</span><span class="o">(</span><span class="s">"violations"</span><span class="o">,</span> <span class="n">violations</span><span class="o">)</span>
        <span class="o">.</span><span class="na">build</span><span class="o">();</span>

<span class="k">return</span> <span class="nc">Response</span><span class="o">.</span><span class="na">status</span><span class="o">(</span><span class="mi">400</span><span class="o">)</span>
        <span class="o">.</span><span class="na">type</span><span class="o">(</span><span class="s">"application/problem+json"</span><span class="o">)</span>
        <span class="o">.</span><span class="na">entity</span><span class="o">(</span><span class="n">problem</span><span class="o">).</span><span class="na">build</span><span class="o">();</span>
</code></pre></div></div>

<p><strong>&#9989; Pros:</strong></p>

<ul>
  <li><strong>Cross-runtime.</strong> Works on Quarkus, Helidon, and Open Liberty &#8211; the same artifact deploys on all three.</li>
  <li><strong>Used in production at Zalando</strong> (and elsewhere); the model handles <code class="language-plaintext highlighter-rouge">cause</code> chains, stack-trace processing, and a few edge cases you probably would not have thought of upfront.</li>
  <li><strong>Jackson integration is done for you</strong> via <code class="language-plaintext highlighter-rouge">jackson-datatype-problem</code>.</li>
</ul>

<p><strong>&#10060; Cons:</strong></p>

<ul>
  <li>One more dependency to track and upgrade.</li>
  <li>You still write the <code class="language-plaintext highlighter-rouge">ExceptionMapper</code>s &#8211; the library standardises the <em>payload</em>, not the <em>wiring</em>.</li>
  <li>If your stack is JSON-B rather than Jackson, you have a bit of extra work.</li>
</ul>

<hr />

<h3 id="3-quarkus-quarkus-http-problem">3. Quarkus: <code class="language-plaintext highlighter-rouge">quarkus-http-problem</code></h3>

<p>If you&#8217;re <em>only</em> targeting Quarkus, the <a href="https://github.com/quarkiverse/quarkus-http-problem"><code class="language-plaintext highlighter-rouge">quarkus-http-problem</code></a> Quarkiverse extension is the shortest path. It auto-maps <code class="language-plaintext highlighter-rouge">ConstraintViolationException</code>, <code class="language-plaintext highlighter-rouge">WebApplicationException</code>, and uncaught <code class="language-plaintext highlighter-rouge">Throwable</code> to <code class="language-plaintext highlighter-rouge">application/problem+json</code> with no boilerplate from you.</p>

<p><strong>&#9989; Pros:</strong></p>

<ul>
  <li>Add the dependency and you get Problem Details for exceptions. No need to write a mapper for each of them.</li>
  <li>Reasonable defaults for validation and security exceptions.</li>
</ul>

<p><strong>&#10060; Cons:</strong></p>

<ul>
  <li><strong>Quarkus only.</strong> Doesn&#8217;t help on Helidon (Jersey) or Open Liberty (CXF). If &#8220;runs on every Jakarta runtime&#8221; is a requirement, this is out.</li>
  <li>Less visibility into <em>what</em> gets mapped to <em>what</em> &#8211; fine until you need to override a default.</li>
</ul>

<hr />

<h3 id="4-spring-boot--a-short-note">4. Spring Boot &#8211; a short note</h3>

<p>For completeness, we need to mention Spring Boot 3+ as well, which has Problem Details built in as <a href="https://docs.spring.io/spring-framework/docs/current/javadoc-api/org/springframework/http/ProblemDetail.html"><code class="language-plaintext highlighter-rouge">org.springframework.http.ProblemDetail</code></a>, with content negotiation and <code class="language-plaintext highlighter-rouge">@ExceptionHandler</code> integration already wired up. If you&#8217;re on Spring, just use it. The JSON structure is the same RFC 9457; only the wiring differs.</p>

<hr />

<h2 id="conclusion">Conclusion</h2>

<p>The point of RFC 9457 is not that there&#8217;s one correct implementation &#8211; there are several reasonable ones &#8211; but that there&#8217;s one correct envelope. Once your API speaks <code class="language-plaintext highlighter-rouge">application/problem+json</code>, clients stop hand-coding error parsers for each new service they consume.</p>

<p>A few rules of thumb:</p>

<ul>
  <li>On <strong>Spring</strong>, use the built-in <code class="language-plaintext highlighter-rouge">ProblemDetail</code>.</li>
  <li>On <strong>Quarkus only</strong>, reach for <code class="language-plaintext highlighter-rouge">quarkus-http-problem</code> and move on.</li>
  <li>For <strong>cross-runtime Jakarta</strong>, choose between <strong>Zalando Problem</strong> (one dependency, more handled for you) and the <strong>hand-made</strong> approach (no dependencies, about 30 lines you fully understand).</li>
</ul>

<p>I picked the hand-made approach for the demo project because portability across Quarkus, Helidon, and Open Liberty mattered, and because the <code class="language-plaintext highlighter-rouge">ExceptionMapper</code> <em>is</em> the demo &#8211; hiding it behind a library would have defeated the point of the talk.</p>

<p>However, &#8220;hand-made&#8221; doesn&#8217;t have to mean &#8220;everyone reinvents it from scratch&#8221;. Write it once, put it in a small internal library, and reuse it across services. That&#8217;s still less code than wiring up a third-party dependency in each runtime.</p>

<h3 id="summary-comparison">Summary Comparison</h3>

<table class="bordered-table">
  <thead>
    <tr>
      <th>Option</th>
      <th>What it gives you</th>
      <th>Runtimes</th>
      <th>Dependency cost</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td><strong>Hand-made</strong> <em>(this post)</em></td>
      <td>~30-line <code class="language-plaintext highlighter-rouge">ProblemDetail</code> + one mapper per error category.</td>
      <td>&#160;&#9989; Quarkus &#160;&#9989; Helidon &#160;&#9989; Open Liberty</td>
      <td>None</td>
    </tr>
    <tr>
      <td><strong>Zalando Problem</strong></td>
      <td><code class="language-plaintext highlighter-rouge">Problem</code> / <code class="language-plaintext highlighter-rouge">ThrowableProblem</code> types + Jackson serialization. You still write the mappers.</td>
      <td>&#160;&#9989; Quarkus &#160;&#9989; Helidon &#160;&#9989; Open Liberty</td>
      <td>1&#8211;2 artifacts</td>
    </tr>
    <tr>
      <td><strong><code class="language-plaintext highlighter-rouge">quarkus-http-problem</code></strong></td>
      <td>Auto-maps validation, <code class="language-plaintext highlighter-rouge">WebApplicationException</code>, and uncaught <code class="language-plaintext highlighter-rouge">Throwable</code>. No boilerplate.</td>
      <td>&#160;&#9989; Quarkus only</td>
      <td>1 extension</td>
    </tr>
    <tr>
      <td><strong>Spring <code class="language-plaintext highlighter-rouge">ProblemDetail</code></strong></td>
      <td>Built into the framework. Content negotiation and <code class="language-plaintext highlighter-rouge">@ExceptionHandler</code> integration.</td>
      <td>&#160;&#9989; Spring Boot 3+</td>
      <td>None (built in)</td>
    </tr>
  </tbody>
</table>

<h2 id="whats-next">What&#8217;s Next?</h2>

<p>Error handling is one of the bonus topics in the <a href="https://github.com/mehmandarov/api-guide-java">API Guide for Java</a>. The same repo also covers OpenAPI documentation, security (RBAC, JWT), pagination, async, and versioning strategies &#8211; see my earlier post on <a href="/api-versioning/">API versioning in Java using JAX-RS</a>.</p>

<p><strong><em>Happy shipping of well-formed error messages, folks!</em></strong></p>

<hr />]]></content><author><name>Rustam Mehmandarov</name></author><summary type="html">A practical look at RFC 9457 Problem Details for HTTP APIs in Jakarta EE &#8211; a hand-made ProblemDetail + ExceptionMapper approach, the Zalando Problem library, and a short note on Quarkus and Spring.</summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://mehmandarov.com/assets/images/posts-images/error.jpg"/><category term="blog"/><category term="english"/><category term="java"/><category term="architecture"/><category term="api"/><category term="jakarta ee"/><category term="microprofile"/><category term="jax-rs"/><category term="error handling"/><category term="quarkus"/><category term="spring"/></entry><entry><title type="html">API versioning in Java using JAX-RS with Jakarta EE and MicroProfile</title><link href="https://mehmandarov.com/api-versioning/" rel="alternate" type="text/html" title="API versioning in Java using JAX-RS with Jakarta EE and MicroProfile"/><published>2026-04-19T09:50:00+02:00</published><updated>2026-04-19T09:50:00+02:00</updated><id>https://mehmandarov.com/api-versioning</id><content type="html" xml:base="https://mehmandarov.com/api-versioning/"><![CDATA[<p><em>Creating APIs and maintaining them over time means often that we need to version them. We will be looking into several ways of doing so in Java using JAX-RS, while building our API end-points using Jakarta EE and MicroProfile. This post was inspired by my talk &#8220;API = Some REST and HTTP, right? RIGHT?!&#8221;</em></p>

<ul>
  <li><a href="#introduction">Introduction</a></li>
  <li><a href="#why-versioning">Why Versioning?</a></li>
  <li><a href="#show-me-the-code">Show Me The CODE!</a></li>
  <li><a href="#1-url-versioning">1. URL Versioning</a></li>
  <li><a href="#2-header-versioning">2. Header Versioning</a></li>
  <li><a href="#3-media-type-versioning-content-negotiation">3. Media Type Versioning</a></li>
  <li><a href="#4-request-parameter-versioning">4. Request Parameter Versioning</a></li>
  <li><a href="#5-bonus-combining-strategies---transparent-uri-rewriting-enterprise-pattern">5. Bonus: Combining Strategies</a></li>
  <li><a href="#6-end-point-deprecation">6. End-Point Deprecation</a></li>
  <li><a href="#summary-comparison">Summary Comparison</a></li>
  <li><a href="#conclusion">Conclusion</a></li>
  <li><a href="#whats-next">What&#8217;s Next?</a></li>
</ul>

<hr />

<h2 id="introduction">Introduction</h2>

<p>When working with APIs over time we would often need to make some changes to end-point definitions &#8211; like adding or deleting resources or changing the attributes for a resource. To ensure backwards compatibility, we often have to introduce <em>versioning</em> for our APIs. APIs, like all software, evolve. You might be adding optional fields or introducing a breaking change. At some point, you will need versioning to support coexistence of the old and new consumers.</p>

<p>However, versioning the API endpoint introduces a question of how this should be done. In this post, we&#8217;ll explore <strong>several common API versioning strategies</strong>, using Jakarta EE and Java.</p>

<blockquote>
  <p>&#128161; Note: There is no silver bullet &#8211; instead, we&#8217;ll explore <strong>pros, cons, and real-world fit</strong>.</p>
</blockquote>

<h2 id="why-versioning">Why Versioning?</h2>

<p>Why not just change the API?<br />
Because breaking contracts is dangerous &#8212; clients may not update in sync, and you&#8217;ll break production consumers.</p>

<p>Versioning allows you to:</p>
<ul>
  <li>Support legacy clients</li>
  <li>Introduce new features safely</li>
  <li>Deprecate responsibly</li>
</ul>

<blockquote>
  <p>&#9888;&#65039; <strong>Caution</strong>: Versioning can cause &#8220;version explosion.&#8221; Each version increases long-term maintenance cost &#8211; aka <em>technical debt</em>.</p>
</blockquote>

<p><strong>Best Practice</strong>: Prefer <em>backward-compatible changes</em> (e.g., adding fields) whenever possible. To mitigate risks, it&#8217;s important to follow best practices for versioning and provide clear documentation and migration paths for users. Also, remember to <em>deprecate</em> old versions to minimize maintenance efforts.</p>

<h2 id="show-me-the-code">Show me the CODE!</h2>

<p>I have created a repository called <a href="https://github.com/mehmandarov/randomstrings/">Random Strings</a> to showcase various concepts. For this blogpost, I would recommend having a look at <a href="https://github.com/mehmandarov/randomstrings/blob/master/src/main/java/com/mehmandarov/randomstrings/apidemo/RandomStringsAPIDemoController.java"><code class="language-plaintext highlighter-rouge">RandomStringsAPIDemoController.java</code></a> and <a href="https://github.com/mehmandarov/randomstrings/blob/master/request_examples.http"><code class="language-plaintext highlighter-rouge">request_examples.http</code></a>. You will find all the info on building and running the code in the repo&#8217;s <code class="language-plaintext highlighter-rouge">README.md</code> file. Each section below will contain &#8220;How to call it&#8221; part with an example using <code class="language-plaintext highlighter-rouge">curl</code> or HTTP-files, and will be based on this repo.</p>

<hr />

<h2 id="1-url-versioning">1. URL Versioning</h2>

<h3 id="what-it-looks-like">What it looks like</h3>
<p>A version appears directly in the URI path. If your API is at <code class="language-plaintext highlighter-rouge">https://example.com/api</code>, and the current version is version 1, the URL for a resource might look like this: <code class="language-plaintext highlighter-rouge">https://example.com/api/v1/resource</code>:</p>

<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nd">@GET</span>
<span class="nd">@Path</span><span class="o">(</span><span class="s">"/v2/"</span><span class="o">)</span>
<span class="nd">@Produces</span><span class="o">(</span><span class="nc">MediaType</span><span class="o">.</span><span class="na">APPLICATION_JSON</span><span class="o">)</span>
<span class="kd">public</span> <span class="nc">Response</span> <span class="nf">getV2</span><span class="o">()</span> <span class="o">{</span>
    <span class="k">return</span> <span class="nc">Response</span><span class="o">.</span><span class="na">status</span><span class="o">(</span><span class="nc">Response</span><span class="o">.</span><span class="na">Status</span><span class="o">.</span><span class="na">NOT_IMPLEMENTED</span><span class="o">)</span>
        <span class="o">.</span><span class="na">entity</span><span class="o">(</span><span class="s">"This v2 using *path versioning* of the API is not implemented."</span><span class="o">)</span>
        <span class="o">.</span><span class="na">build</span><span class="o">();</span>
<span class="o">}</span>
</code></pre></div></div>

<h3 id="how-to-call-it">How to call it</h3>

<p><strong>cURL:</strong></p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>curl <span class="nt">-X</span> GET http://localhost:8080/api/rnd/v2/ <span class="se">\</span>
  <span class="nt">-H</span> <span class="s2">"Accept: application/json"</span>
</code></pre></div></div>

<p><strong>HTTP Request (<code class="language-plaintext highlighter-rouge">.http</code> file):</strong></p>
<div class="language-http highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="err">GET http://localhost:8080/api/rnd/v2/
Accept: application/json
</span></code></pre></div></div>

<p><strong>&#9989; Pros:</strong></p>
<ul>
  <li>Simple and intuitive. Visible.</li>
  <li>Easy to test (e.g., with curl or Postman directly in a browser).</li>
  <li>Plays well with gateways and reverse proxies.</li>
  <li>Clear visual distinction between versions.</li>
</ul>

<p><strong>&#10060; Cons:</strong></p>
<ul>
  <li>Pollutes the URI with versioning logic.</li>
  <li>Breaks REST&#8217;s principle of stable resource identifiers.</li>
  <li>Clients have to update URLs when migrating.</li>
  <li>Risk of accumulating too many legacy versions.</li>
  <li>Can result in cluttered and difficult-to-read URLs if there are multiple versions of the API.</li>
</ul>

<p><strong>&#128269; However:</strong> Despite its REST purism flaw, URL versioning is extremely practical and widely adopted.</p>

<h2 id="2-header-versioning">2. Header Versioning</h2>

<h3 id="what-it-looks-like-1">What it looks like</h3>
<p>Client specifies version in a custom HTTP header (e.g., <code class="language-plaintext highlighter-rouge">Accept-Version</code>, <code class="language-plaintext highlighter-rouge">X-API-Version</code>, etc.):</p>

<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nd">@Path</span><span class="o">(</span><span class="s">"/hi2"</span><span class="o">)</span>
<span class="nd">@GET</span>
<span class="nd">@Produces</span><span class="o">({</span><span class="s">"application/json"</span><span class="o">})</span>
<span class="kd">public</span> <span class="nc">String</span> <span class="nf">entryPoint2</span><span class="o">(</span><span class="nd">@HeaderParam</span><span class="o">(</span><span class="s">"Accept-Version"</span><span class="o">)</span> <span class="nc">String</span> <span class="n">apiVersion</span><span class="o">)</span> <span class="o">{</span>
    <span class="k">if</span> <span class="o">(</span><span class="n">apiVersion</span> <span class="o">==</span> <span class="kc">null</span> <span class="o">||</span> <span class="n">apiVersion</span><span class="o">.</span><span class="na">isEmpty</span><span class="o">())</span> <span class="o">{</span>
        <span class="k">return</span> <span class="s">"Default unversioned endpoint hit."</span><span class="o">;</span>
    <span class="o">}</span>
    <span class="nc">String</span> <span class="n">message</span> <span class="o">=</span> <span class="s">"Versioned: Using custom headers. Using version: "</span> <span class="o">+</span> <span class="n">apiVersion</span> <span class="o">+</span><span class="s">"."</span><span class="o">;</span>
    <span class="k">return</span> <span class="n">message</span><span class="o">;</span>
<span class="o">}</span>
</code></pre></div></div>

<h3 id="how-to-call-it-1">How to call it</h3>

<p><em>Note: This is for demo purposes only. It has to have a different URL than the regular API; otherwise, it will also intercept calls that do not contain the <code class="language-plaintext highlighter-rouge">Accept-Version</code> header.</em></p>

<p><strong>cURL:</strong></p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>curl <span class="nt">-X</span> GET http://localhost:8080/api/rnd/versioned/ <span class="se">\</span>
  <span class="nt">-H</span> <span class="s2">"Accept: application/json"</span> <span class="se">\</span>
  <span class="nt">-H</span> <span class="s2">"Accept-Version: 2"</span>
</code></pre></div></div>

<p><strong>HTTP Request (<code class="language-plaintext highlighter-rouge">.http</code> file):</strong></p>
<div class="language-http highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="err">GET http://localhost:8080/api/rnd/versioned/
Accept: application/json
Accept-Version: 2
</span></code></pre></div></div>

<p><strong>&#9989; Pros:</strong></p>
<ul>
  <li>Keeps URL structure clean and predictable.</li>
  <li>Closer to HTTP semantics (headers = metadata).</li>
  <li>Allows centralized versioning logic in filters/interceptors.</li>
</ul>

<p><strong>&#10060; Cons:</strong></p>
<ul>
  <li>Not self-descriptive &#8212; clients must &#8220;know the secret handshake&#8221;.</li>
  <li>Poor discoverability (not visible in browser without tools).</li>
  <li>Breaks caching in some proxies/CDNs unless explicitly configured.</li>
  <li>Adds complexity to tooling and testing.</li>
</ul>

<p><strong>&#9888;&#65039; Challenge:</strong> Header versioning can feel &#8220;invisible&#8221; and cause developer confusion if not well-documented.</p>

<h2 id="3-media-type-versioning-content-negotiation">3. Media Type Versioning (Content Negotiation)</h2>

<h3 id="what-it-looks-like-2">What it looks like</h3>
<p>Client specifies version via a custom media type in the <code class="language-plaintext highlighter-rouge">Accept</code> header. This is sometimes called <a href="https://developer.mozilla.org/en-US/docs/Web/HTTP/Content_negotiation">Content Negotiation</a> versioning.</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Accept: application/hi.v3+json
</code></pre></div></div>

<p>In Jakarta EE:</p>

<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nd">@Path</span><span class="o">(</span><span class="s">"/hi"</span><span class="o">)</span>
<span class="nd">@GET</span>
<span class="nd">@Produces</span><span class="o">({</span><span class="s">"application/hi.v3+json"</span><span class="o">,</span> <span class="s">"application/hi.v4+json"</span><span class="o">})</span>
<span class="kd">public</span> <span class="nc">String</span> <span class="nf">entryPoint</span><span class="o">()</span> <span class="kd">throws</span> <span class="nc">URISyntaxException</span> <span class="o">{</span>
    <span class="nc">String</span> <span class="n">message</span> <span class="o">=</span> <span class="s">"Versioned: Hai there!"</span><span class="o">;</span>
    <span class="k">return</span> <span class="n">message</span><span class="o">;</span>
<span class="o">}</span>
</code></pre></div></div>

<h3 id="how-to-call-it-2">How to call it</h3>

<p>You can request different versions (e.g., v3, v4, v5) by updating the media type:</p>

<p><strong>cURL:</strong></p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>curl <span class="nt">-X</span> GET http://localhost:8080/api/rnd/ <span class="se">\</span>
  <span class="nt">-H</span> <span class="s2">"Accept: application/rnd.v3+json"</span>
</code></pre></div></div>

<p><strong>HTTP Request (<code class="language-plaintext highlighter-rouge">.http</code> file):</strong></p>
<div class="language-http highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="err">GET http://localhost:8080/api/rnd/
Accept: application/rnd.v3+json
</span></code></pre></div></div>

<p><strong>&#9989; Pros:</strong></p>
<ul>
  <li>Very REST-compliant: changes representation, not resource.</li>
  <li>URI remains stable.</li>
  <li>Supports richer format negotiation (e.g., XML, HAL, etc.).</li>
</ul>

<p><strong>&#10060; Cons:</strong></p>
<ul>
  <li>Requires strict control over media types.</li>
  <li>Not all clients/tooling handle custom media types well.</li>
  <li>Breaks with some reverse proxies and middleware that don&#8217;t forward full Accept headers.</li>
  <li>More work to configure content negotiation.</li>
</ul>

<p><strong>&#129514; Observation:</strong> Elegant in design, but rarely used consistently in real-world public APIs.</p>

<h2 id="4-request-parameter-versioning">4. Request Parameter Versioning</h2>

<h3 id="what-it-looks-like-3">What it looks like</h3>
<p><em>Technically</em>, it is also possible for the client to specify the version in a URL query parameter (e.g., <code class="language-plaintext highlighter-rouge">?version=2</code>). This, however, might not be a suggested strategy, in my opinion.</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>https://example.com/api/resource?version<span class="o">=</span>2
</code></pre></div></div>

<h3 id="how-to-call-it-3">How to call it</h3>

<p><strong>cURL:</strong></p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>curl <span class="nt">-X</span> GET http://localhost:8080/api/rnd?version<span class="o">=</span>2 <span class="se">\</span>
  <span class="nt">-H</span> <span class="s2">"Accept: application/json"</span>
</code></pre></div></div>

<p><strong>HTTP Request (<code class="language-plaintext highlighter-rouge">.http</code> file):</strong></p>
<div class="language-http highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="err">GET http://localhost:8080/api/rnd?version=2
Accept: application/json
</span></code></pre></div></div>

<p><strong>&#9989; Pros:</strong></p>
<ul>
  <li>Simplicity &amp; discoverability: Easy to test in a browser without specialized tools.</li>
  <li>Defaulting logic: Straightforward to implement &#8220;default to latest&#8221; if the parameter is omitted.</li>
  <li>Caching friendly: CDNs treat different query params as unique resources by default.</li>
</ul>

<p><strong>&#10060; Cons:</strong></p>
<ul>
  <li>URI Pollution: Mixes resource identification with technical metadata.</li>
  <li>Routing complexity: Routing based on query parameters usually requires custom middleware or manual logic inside the controller.</li>
  <li>Harder to generate clean, automated documentation (like OpenAPI) when multiple versions share the same path.</li>
</ul>

<h2 id="5-bonus-combining-strategies---transparent-uri-rewriting-enterprise-pattern">5. Bonus: Combining Strategies - Transparent URI Rewriting (Enterprise Pattern)</h2>

<p>In large enterprises, you might find that different clients have different needs. Some prefer the explicitness of URL versioning, while others require the clean URIs of Header versioning. You don&#8217;t have to choose just one&#8212;you can support both without duplicating your backend routing logic.</p>

<p>The common practice is to structure all your resource classes using <strong>URL versioning</strong> (e.g., <code class="language-plaintext highlighter-rouge">@Path("/v1/resource")</code>), but use a <strong><code class="language-plaintext highlighter-rouge">@PreMatching</code> Filter</strong> to intercept requests and transparently rewrite the URI if a client uses a header instead.</p>

<p>Here is what that looks like in Jakarta EE using a <code class="language-plaintext highlighter-rouge">ContainerRequestFilter</code>:</p>

<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nd">@Provider</span>
<span class="nd">@PreMatching</span>
<span class="kd">public</span> <span class="kd">class</span> <span class="nc">HeaderVersionFilter</span> <span class="kd">implements</span> <span class="nc">ContainerRequestFilter</span> <span class="o">{</span>

      <span class="nd">@Override</span>
      <span class="kd">public</span> <span class="kt">void</span> <span class="nf">filter</span><span class="o">(</span><span class="nc">ContainerRequestContext</span> <span class="n">ctx</span><span class="o">)</span> <span class="o">{</span>
          <span class="nc">String</span> <span class="n">path</span> <span class="o">=</span> <span class="n">ctx</span><span class="o">.</span><span class="na">getUriInfo</span><span class="o">().</span><span class="na">getPath</span><span class="o">();</span>

          <span class="c1">// If the path is already versioned (e.g., starts with v1, v2), let it pass</span>
          <span class="k">if</span> <span class="o">(</span><span class="n">path</span><span class="o">.</span><span class="na">matches</span><span class="o">(</span><span class="s">"v\\d+(/.*)?"</span><span class="o">))</span> <span class="k">return</span><span class="o">;</span>

          <span class="c1">// Otherwise, check if the client provided a version header</span>
          <span class="nc">String</span> <span class="n">version</span> <span class="o">=</span> <span class="n">ctx</span><span class="o">.</span><span class="na">getHeaderString</span><span class="o">(</span><span class="s">"X-API-Version"</span><span class="o">);</span>

          <span class="k">if</span> <span class="o">(</span><span class="n">version</span> <span class="o">!=</span> <span class="kc">null</span> <span class="o">&amp;&amp;</span> <span class="o">!</span><span class="n">version</span><span class="o">.</span><span class="na">isEmpty</span><span class="o">())</span> <span class="o">{</span>
              <span class="c1">// Transparently rewrite the URI internally to match our URL-based routes</span>
              <span class="nc">String</span> <span class="n">newPath</span> <span class="o">=</span> <span class="s">"v"</span> <span class="o">+</span> <span class="n">version</span> <span class="o">+</span> <span class="s">"/"</span> <span class="o">+</span> <span class="n">path</span><span class="o">;</span>
              <span class="no">URI</span> <span class="n">baseUri</span> <span class="o">=</span> <span class="n">ctx</span><span class="o">.</span><span class="na">getUriInfo</span><span class="o">().</span><span class="na">getBaseUri</span><span class="o">();</span>
              <span class="no">URI</span> <span class="n">newUri</span> <span class="o">=</span> <span class="nc">UriBuilder</span><span class="o">.</span><span class="na">fromUri</span><span class="o">(</span><span class="n">baseUri</span><span class="o">).</span><span class="na">path</span><span class="o">(</span><span class="n">newPath</span><span class="o">).</span><span class="na">build</span><span class="o">();</span>

              <span class="n">ctx</span><span class="o">.</span><span class="na">setRequestUri</span><span class="o">(</span><span class="n">baseUri</span><span class="o">,</span> <span class="n">newUri</span><span class="o">);</span>
          <span class="o">}</span>
      <span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>

<p><strong>&#9989; Pros:</strong></p>
<ul>
  <li><strong>Ultimate Flexibility:</strong> Clients can use <code class="language-plaintext highlighter-rouge">http://api.example.com/v2/resource</code> OR <code class="language-plaintext highlighter-rouge">http://api.example.com/resource</code> with an <code class="language-plaintext highlighter-rouge">X-API-Version: 2</code> header.</li>
  <li><strong>Single Source of Truth:</strong> Your backend controllers only need to use <code class="language-plaintext highlighter-rouge">@Path("/v2/")</code>. You don&#8217;t have to write duplicate methods to handle both headers and paths.</li>
</ul>

<p><strong>&#10060; Cons:</strong></p>
<ul>
  <li><strong>Magic Routing:</strong> It introduces a layer of &#8220;magic&#8221; where the requested URI differs from the routed URI, which can briefly confuse new developers debugging the application.</li>
</ul>

<p><strong>&#128161; Want to know more?</strong> Read up on terms <em><code class="language-plaintext highlighter-rouge">Version Normalization</code></em> and <em><code class="language-plaintext highlighter-rouge">Internal Decoupling</code></em>.</p>

<h2 id="6-end-point-deprecation">6. End-Point Deprecation</h2>

<p>Eventually, you will need to retire old API versions. Remember: every old version you keep around is <em>technical debt</em> &#8212; it increases long-term maintenance cost. When deprecating an endpoint, consider the following best practices:</p>

<ol>
  <li><strong>Update the Docs:</strong> Use OpenAPI&#8217;s <code class="language-plaintext highlighter-rouge">@Operation</code> annotation to clearly mark it as deprecated.</li>
  <li><strong>Add <code class="language-plaintext highlighter-rouge">@Deprecated</code>:</strong> Use the Java <code class="language-plaintext highlighter-rouge">@Deprecated</code> annotation where necessary.</li>
  <li><strong>HTTP Redirects:</strong> Consider returning HTTP codes like <code class="language-plaintext highlighter-rouge">302 Found</code> or <code class="language-plaintext highlighter-rouge">301 Moved Permanently</code> after some time.</li>
  <li><strong>Add a Link header:</strong> Provide a link to the new version in the response headers.</li>
  <li><strong>Log / Count calls:</strong> Track usage (e.g., with MicroProfile <code class="language-plaintext highlighter-rouge">@Counted</code>) to know when it is safe to finally remove the endpoint.</li>
</ol>

<p>Here is a practical example in Jakarta EE showing how to deprecate an endpoint, add a <code class="language-plaintext highlighter-rouge">Link</code> header, and track metrics:</p>

<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nd">@GET</span>
<span class="nd">@Path</span><span class="o">(</span><span class="s">"v0.1/"</span><span class="o">)</span>
<span class="nd">@Produces</span><span class="o">(</span><span class="nc">MediaType</span><span class="o">.</span><span class="na">APPLICATION_JSON</span><span class="o">)</span>
<span class="nd">@Operation</span><span class="o">(</span><span class="n">summary</span> <span class="o">=</span> <span class="s">"DEPRECATED. Use v2 now. Returns the adjective-noun pair"</span><span class="o">,</span>
           <span class="n">description</span> <span class="o">=</span> <span class="s">"Deprecated function. The pair of one random adjective and one random noun is returned as an array."</span><span class="o">)</span>
<span class="nd">@Counted</span><span class="o">(</span><span class="n">name</span> <span class="o">=</span> <span class="s">"totalCountToRandomPairCalls_Versioned_Path_DEPRECATED"</span><span class="o">,</span>
         <span class="n">absolute</span> <span class="o">=</span> <span class="kc">true</span><span class="o">,</span>
         <span class="n">description</span> <span class="o">=</span> <span class="s">"Deprecated function call: Total number of calls to random string pairs."</span><span class="o">,</span>
         <span class="n">tags</span> <span class="o">=</span> <span class="o">{</span><span class="s">"calls=pairs"</span><span class="o">})</span>
<span class="nd">@Deprecated</span>
<span class="kd">public</span> <span class="nc">Response</span> <span class="nf">getRndStringPathDeprecated</span><span class="o">()</span> <span class="o">{</span>
    <span class="no">URI</span> <span class="n">newVersionURI</span> <span class="o">=</span> <span class="nc">UriBuilder</span><span class="o">.</span><span class="na">fromUri</span><span class="o">(</span><span class="s">"/api/rnd/v2/"</span><span class="o">).</span><span class="na">build</span><span class="o">();</span>
    <span class="nc">Link</span> <span class="n">newVersionLink</span> <span class="o">=</span> <span class="nc">Link</span><span class="o">.</span><span class="na">fromUri</span><span class="o">(</span><span class="n">newVersionURI</span><span class="o">).</span><span class="na">rel</span><span class="o">(</span><span class="s">"alternate"</span><span class="o">).</span><span class="na">build</span><span class="o">();</span>
    <span class="k">return</span> <span class="nc">Response</span><span class="o">.</span><span class="na">ok</span><span class="o">(</span><span class="s">"Deprecated response"</span><span class="o">,</span> <span class="nc">MediaType</span><span class="o">.</span><span class="na">APPLICATION_JSON</span><span class="o">)</span>
            <span class="o">.</span><span class="na">header</span><span class="o">(</span><span class="n">jakarta</span><span class="o">.</span><span class="na">ws</span><span class="o">.</span><span class="na">rs</span><span class="o">.</span><span class="na">core</span><span class="o">.</span><span class="na">HttpHeaders</span><span class="o">.</span><span class="na">LINK</span><span class="o">,</span> <span class="n">newVersionLink</span><span class="o">.</span><span class="na">toString</span><span class="o">())</span>
            <span class="o">.</span><span class="na">header</span><span class="o">(</span><span class="s">"X-API-Version"</span><span class="o">,</span> <span class="s">"0.1"</span><span class="o">)</span>
            <span class="o">.</span><span class="na">build</span><span class="o">();</span>
<span class="o">}</span>
</code></pre></div></div>

<h3 id="how-to-call-it-deprecated-endpoint">How to call it (Deprecated endpoint)</h3>

<p><strong>cURL:</strong></p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>curl <span class="nt">-X</span> GET http://localhost:8080/api/rnd/v0.1/ <span class="se">\</span>
  <span class="nt">-H</span> <span class="s2">"Accept: application/json"</span>
</code></pre></div></div>

<p><strong>HTTP Request (<code class="language-plaintext highlighter-rouge">.http</code> file):</strong></p>
<div class="language-http highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="err">GET http://localhost:8080/api/rnd/v0.1/
Accept: application/json
</span></code></pre></div></div>

<h2 id="summary-comparison">Summary Comparison</h2>

<p>The following table summarizes all the different routing strategies implemented in the <a href="https://github.com/mehmandarov/randomstrings/">demo project</a>, illustrating how the HTTP method, path, and headers combine to invoke the correct Java method. The method names refer to the methods in <a href="https://github.com/mehmandarov/randomstrings/blob/master/src/main/java/com/mehmandarov/randomstrings/apidemo/RandomStringsAPIDemoController.java"><code class="language-plaintext highlighter-rouge">RandomStringsAPIDemoController.java</code></a> (or <code class="language-plaintext highlighter-rouge">RandomStringsController.java</code>):</p>

<table>
  <thead>
    <tr>
      <th>HTTP Method</th>
      <th>Path</th>
      <th>Headers</th>
      <th>Method Invoked</th>
      <th>Notes</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td><code class="language-plaintext highlighter-rouge">GET</code></td>
      <td><code class="language-plaintext highlighter-rouge">/rnd</code></td>
      <td><em>None</em></td>
      <td><code class="language-plaintext highlighter-rouge">getRndString()</code></td>
      <td>Default (unversioned) endpoint</td>
    </tr>
    <tr>
      <td><code class="language-plaintext highlighter-rouge">GET</code></td>
      <td><code class="language-plaintext highlighter-rouge">/rnd</code></td>
      <td><code class="language-plaintext highlighter-rouge">Accept: application/json</code></td>
      <td><code class="language-plaintext highlighter-rouge">getRndString()</code></td>
      <td>Standard media type</td>
    </tr>
    <tr>
      <td><code class="language-plaintext highlighter-rouge">GET</code></td>
      <td><code class="language-plaintext highlighter-rouge">/rnd/v2/</code></td>
      <td><em>Any</em></td>
      <td><code class="language-plaintext highlighter-rouge">getRndStringV2path()</code></td>
      <td>Demo for <strong>path-based</strong> versioning</td>
    </tr>
    <tr>
      <td><code class="language-plaintext highlighter-rouge">GET</code></td>
      <td><code class="language-plaintext highlighter-rouge">/rnd/versioned</code></td>
      <td><em>None</em></td>
      <td><code class="language-plaintext highlighter-rouge">getRndStringV2Header()</code></td>
      <td>Fallback to <code class="language-plaintext highlighter-rouge">getRndString()</code> if header is missing</td>
    </tr>
    <tr>
      <td><code class="language-plaintext highlighter-rouge">GET</code></td>
      <td><code class="language-plaintext highlighter-rouge">/rnd/versioned</code></td>
      <td><code class="language-plaintext highlighter-rouge">Accept-Version: 2</code></td>
      <td><code class="language-plaintext highlighter-rouge">getRndStringV2Header()</code></td>
      <td><strong>Header-based</strong> versioning</td>
    </tr>
    <tr>
      <td><code class="language-plaintext highlighter-rouge">GET</code></td>
      <td><code class="language-plaintext highlighter-rouge">/rnd</code></td>
      <td><code class="language-plaintext highlighter-rouge">Accept: application/rnd.v3+json</code></td>
      <td><code class="language-plaintext highlighter-rouge">getRndStringV3V4MediaType()</code></td>
      <td><strong>Media type versioning</strong> &#8212; v3</td>
    </tr>
    <tr>
      <td><code class="language-plaintext highlighter-rouge">GET</code></td>
      <td><code class="language-plaintext highlighter-rouge">/rnd</code></td>
      <td><code class="language-plaintext highlighter-rouge">Accept: application/rnd.v4+json</code></td>
      <td><code class="language-plaintext highlighter-rouge">getRndStringV3V4MediaType()</code></td>
      <td><strong>Media type versioning</strong> &#8212; v4</td>
    </tr>
    <tr>
      <td><code class="language-plaintext highlighter-rouge">GET</code></td>
      <td><code class="language-plaintext highlighter-rouge">/rnd</code></td>
      <td><code class="language-plaintext highlighter-rouge">Accept: application/rnd.v5+json</code></td>
      <td><code class="language-plaintext highlighter-rouge">getRndStringV5MediaType()</code></td>
      <td><strong>Media type versioning</strong> &#8212; v5</td>
    </tr>
  </tbody>
</table>

<h2 id="conclusion">Conclusion</h2>

<p>There is no single correct approach to API versioning. For most teams and public APIs, <strong>URL versioning</strong> is good enough&#8212;it&#8217;s visible, easy to test, and plays well with existing tooling. However, you might want to use <strong>header versioning</strong> if your APIs are primarily consumed by internal services or SDKs that can abstract away the complexity. Reserve <strong>media type versioning</strong> for hypermedia-rich or REST-purist APIs, and only if your tooling supports it end-to-end.</p>

<p>Consider who your consumers are, whether your API is public or internal, your infrastructure maturity, and your team&#8217;s ability to support multiple versions.</p>

<h2 id="whats-next">What&#8217;s Next?</h2>

<p>Versioning is just one part of building robust REST APIs. If you want to dive deeper, have a look at the <a href="https://github.com/mehmandarov/api-guide-java">API Guide for Java</a> repository and the slides in the <a href="https://github.com/mehmandarov/api-guide-java/tree/main/presentation">presentation folder</a>. They cover documentation with OpenAPI, security best practices (like RBAC and JWT integration), advanced patterns (pagination, async APIs), and going beyond REST with gRPC and GraphQL.</p>

<p><strong><em>Happy coding!</em></strong></p>

<hr />]]></content><author><name>Rustam Mehmandarov</name></author><summary type="html">Exploring common API versioning strategies in Java using JAX-RS with Jakarta EE and MicroProfile &#8211; URL, header, and media type versioning &#8211; with pros, cons, and practical code examples.</summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://mehmandarov.com/assets/images/posts-images/container-ship.jpeg"/><category term="blog"/><category term="english"/><category term="java"/><category term="architecture"/><category term="api"/><category term="jakarta ee"/><category term="microprofile"/><category term="jax-rs"/><category term="openapi"/></entry><entry><title type="html">Multiplying The Developer Joy: Multiple Quarkus Containers + Simultaneous Remote Development Sessions</title><link href="https://mehmandarov.com/remote-dev-mode-quarkus/" rel="alternate" type="text/html" title="Multiplying The Developer Joy: Multiple Quarkus Containers + Simultaneous Remote Development Sessions"/><published>2024-03-30T10:50:00+01:00</published><updated>2024-03-30T10:50:00+01:00</updated><id>https://mehmandarov.com/remote-dev-mode-quarkus</id><content type="html" xml:base="https://mehmandarov.com/remote-dev-mode-quarkus/"><![CDATA[<p><em>Running several simultaneous Quarkus app containers on your machine with the Quarkus&#8217; remote development mode activated presents some challenges. Let&#8217;s have a look at how we can fix this.</em></p>

<ul>
  <li><a href="#introduction">Introduction</a></li>
  <li><a href="#setup">Setup</a></li>
  <li><a href="#conclusion">Conclusion</a></li>
</ul>

<h2 id="introduction">Introduction</h2>

<p>The built-in <a href="https://quarkus.io/guides/maven-tooling#dev-mode/">development mode</a> for Quarkus is a great functionality that lets you update the application code, resources, and configurations. Setting it up is a great way to develop your applications <em>locally</em>, as you can immediately see the changes reflected in your application.</p>

<p>Furthermore, we have a <a href="https://quarkus.io/guides/maven-tooling#remote-development-mode">remote development mode</a>, which lets you make changes to local files immediately available in a containerized environment. Remote development mode works excellently if the container runs in a local Docker or remote containerized environment.</p>

<p>However, running several simultaneous containers with the remote development mode on, mapped to the same domain, may result in warnings and erratic behavior from the client side.</p>

<h2 id="setup">Setup</h2>

<p>Imagine a setup where you are running a set of containers, for example, using <code class="language-plaintext highlighter-rouge">docker-compose</code> and mapping them all to <code class="language-plaintext highlighter-rouge">my.cluster.host.com</code> (or even <code class="language-plaintext highlighter-rouge">localhost</code>)through several ports:</p>

<p><img src="/assets/images/posts-images/2024-03-30-microservices.png" alt="Microservice Setup" /></p>
<figcaption class="caption">Microservice Setup Example</figcaption>

<p>First, you will need to update <code class="language-plaintext highlighter-rouge">quarkus.live-reload.url</code> in the properties for all the apps (see <a href="https://quarkus.io/guides/maven-tooling#remote-development-mode">docs</a> on where and how to do this). Update the settings to the correct domain and port (in our case, it is <code class="language-plaintext highlighter-rouge">8080</code>, <code class="language-plaintext highlighter-rouge">8081</code>, or <code class="language-plaintext highlighter-rouge">8082</code>):</p>

<div class="language-properties highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="py">quarkus.live-reload.url</span><span class="p">=</span><span class="s">http://localhost:8081</span>
</code></pre></div></div>

<p>Next, try starting your containers with the remote development mode enabled and connect to the application from a terminal or an IDE. For the second and the consecutive applications, the attempts to establish a connection you will see the following message in the logs:</p>

<pre><code class="language-commandline">$&gt; ./mvnw quarkus:remote-dev -Dquarkus.profile=dev

&lt; ... &gt;

[WARNING] Changed debug port to 57409 because of a port conflict
Listening for transport dt_socket at address: 57409

&lt; ... &gt;
</code></pre>

<p><em><strong>Note:</strong> Fallback ports will be random and may vary from the one above.</em></p>

<p>This setup will break the remote reloading from the terminal on the client side (i.e., your IDE). Two or more of your client applications now see that the default port <code class="language-plaintext highlighter-rouge">50005</code> for a remote debug is in use and start with a new, random port.</p>

<p>The simple fix is to update the debug ports for all other applications to something other than <code class="language-plaintext highlighter-rouge">5005</code>, such as <code class="language-plaintext highlighter-rouge">6006</code> and <code class="language-plaintext highlighter-rouge">6007</code>. Custom debug ports can be set in the <code class="language-plaintext highlighter-rouge">pom.xml</code> files, under <code class="language-plaintext highlighter-rouge">quarkus-maven-plugin</code>, for each of the applications that require this update:</p>

<div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt">&lt;build&gt;</span>
  <span class="nt">&lt;plugins&gt;</span>
    <span class="nt">&lt;plugin&gt;</span>
      <span class="nt">&lt;groupId&gt;</span>io.quarkus.platform<span class="nt">&lt;/groupId&gt;</span>
      <span class="nt">&lt;artifactId&gt;</span>quarkus-maven-plugin<span class="nt">&lt;/artifactId&gt;</span>
      <span class="nt">&lt;version&gt;</span>${quarkus.platform.version}<span class="nt">&lt;/version&gt;</span>
      <span class="c">&lt;!-- ADD THE CONFIGURATION MENTIONED BELOW THIS LINE --&gt;</span>
      <span class="nt">&lt;configuration&gt;</span>
        <span class="nt">&lt;debug&gt;</span>6006<span class="nt">&lt;/debug&gt;</span>
      <span class="nt">&lt;/configuration&gt;</span>
...
</code></pre></div></div>

<p>You can choose whether to update the debug ports for all applications in the cluster or for all applications except one, which will get the default port.</p>

<p>Now, you will need to rebuild your apps and re-initiate the remote development mode for each container. And, voil&#224;, everything works!</p>

<p><strong><em>One last note</em></strong>: Please ensure you do not use the remote development functionality in the production environment.</p>

<h2 id="conclusion">Conclusion</h2>
<p>A tiny config update brings back the development joy of using remote development mode for more than one container simultaneously.</p>

<p><strong><em>Happy coding!</em></strong></p>]]></content><author><name>Rustam Mehmandarov</name></author><summary type="html">Running several simultaneous Quarkus app containers on your machine with the Quarkus&#8217; remote development mode activated presents some challenges. Let&#8217;s have a look at how we can fix this.</summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://mehmandarov.com/assets/images/posts-images/container-ship.jpeg"/><category term="blog"/><category term="english"/><category term="java"/><category term="containers"/><category term="quarkus"/><category term="docker"/></entry><entry><title type="html">Cloud Security Principles: Part 2</title><link href="https://mehmandarov.com/cloud-security-principles-2/" rel="alternate" type="text/html" title="Cloud Security Principles: Part 2"/><published>2023-10-17T15:50:00+02:00</published><updated>2023-10-17T15:50:00+02:00</updated><id>https://mehmandarov.com/cloud-security-principles-2</id><content type="html" xml:base="https://mehmandarov.com/cloud-security-principles-2/"><![CDATA[<p><em>This is the second part of the series on the Cloud Security Principles. This post will look at some key principles for securing your applications. Similarly to the <a href="/cloud-security-principles/">first post</a>, some prior knowledge of various IT architecture and security concepts may be expected. This post was inspired by a talk I have recently done with <a href="https://linktr.ee/nehasardana">Neha Sardana</a> at JAX New York.</em></p>

<ul>
  <li><a href="#introduction">Introduction</a></li>
  <li><a href="#principles">Principles</a></li>
  <li><a href="#conclusion">Conclusion</a></li>
</ul>

<hr />

<h2 id="introduction">Introduction</h2>
<p><a href="/cloud-security-principles/">In the first part</a>, we have summed up all the essential elements to consider when working with Cloud and securing cloud-native applications/platforms. In this post, we would like to give you some concrete principles and tips for creating more secure applications.</p>

<h2 id="principles">Principles</h2>
<h3 id="multi-layered-defense">Multi-Layered Defense</h3>
<p><em><strong>Keywords</strong></em>: <code class="language-plaintext highlighter-rouge">general</code></p>

<p>First of all, a more generic but important principle: It would be best to look at security as a whole &#8211; integrating various security layers on multiple levels in any system. It should include cyber-security plans for:</p>

<ol>
  <li>Devices</li>
  <li>Applications</li>
  <li>Networks</li>
  <li>Infrastructure</li>
  <li>People</li>
</ol>

<p>Think of this principle as all the layers of clothing you wear to protect yourself from cold and bad weather. If one of the layers is compromised, there is always another to keep you warm and dry.</p>

<h3 id="identity-and-access-management-iam-misconfiguration">Identity and Access Management (IAM) Misconfiguration</h3>
<p><em><strong>Keywords</strong></em>: <code class="language-plaintext highlighter-rouge">network</code>, <code class="language-plaintext highlighter-rouge">permissions</code></p>

<p>You need to control access and permissions meticulously and over time.
Things to consider:</p>
<ul>
  <li>Implementing role-based access control (RBAC)</li>
  <li>Principle of least privilege</li>
  <li>Routines for updating and removing permissions when they are no longer needed.</li>
  <li>Explore possibilities for using time-based conditions for IAM policies.</li>
</ul>

<h3 id="api-security">API Security</h3>
<p><em><strong>Keywords</strong></em>: <code class="language-plaintext highlighter-rouge">endpoints</code>, <code class="language-plaintext highlighter-rouge">permissions</code></p>

<ul>
  <li>APIs act as the gateways to your application and data. Securing access to and securing them from known vulnerabilities is paramount to prevent unauthorized access and data breaches.</li>
  <li>Utilize <em>authentication</em>, <em>authorization</em>, and <em>API gateways</em> to control access and protect sensitive information. Don&#8217;t forget to monitor the software or libraries that make APIs available (e.g., runtimes, middleware)</li>
</ul>

<h3 id="data-encryption">Data Encryption</h3>
<p><em><strong>Keywords</strong></em>: <code class="language-plaintext highlighter-rouge">data</code></p>

<ul>
  <li>Safeguarding data at rest, in transit, and during processing is critical for your applications.</li>
  <li>Utilize encryption, tokenization, and data masking techniques to ensure data protection. Removing unnecessary sensitive information can simplify some of these tasks.</li>
  <li>If a platform or a Cloud provider provides the encryption, consider if you would like to use the standard keys for encryption or &#8220;bring your own&#8221; and manage them yourself or through a third party.</li>
  <li>Beware: Don&#8217;t write your own crypto! Ever.</li>
</ul>

<h3 id="zero-trust">Zero Trust</h3>
<p><em><strong>Keywords</strong></em>: <code class="language-plaintext highlighter-rouge">network</code>, <code class="language-plaintext highlighter-rouge">permissions</code></p>

<ul>
  <li>The Zero Trust security model assumes that no one is inherently trustworthy, even those within your network.</li>
  <li>This is opposed to more traditional approaches where perimeter security was prioritized over security inside the network.</li>
  <li>Adopting this approach, every request, user, and device is thoroughly verified before gaining access.</li>
  <li>Again: Implement the principle of least privilege, where users are only granted the minimum level of access required to perform their tasks.</li>
</ul>

<h3 id="software-supply-chain-security">Software Supply Chain Security</h3>
<p><em><strong>Keywords</strong></em>: <code class="language-plaintext highlighter-rouge">software</code>, <code class="language-plaintext highlighter-rouge">environment</code></p>

<ul>
  <li>Create <a href="https://www.cisa.gov/sbom">Software Bill of Materials (SBOM)</a> for your software</li>
  <li>Governance: Know where all the building blocks (artifacts) of your software are coming from.</li>
  <li>Automate security checks within your CI/CD pipeline to catch vulnerabilities early and often.</li>
  <li>Use static code analysis with tools like SonarQube to scan your code for potential security flaws and integrate those checks into your CI/CD pipeline to ensure continuous security monitoring.</li>
  <li>Use tools to monitor not only the code you develop yourself but also all the third-party libraries you utilize in your code.</li>
  <li>With DevSecOps, automated security security is becoming integral to the development process. Adopt it if you haven&#8217;t done so already.</li>
</ul>

<h3 id="secure-containerization">Secure Containerization</h3>
<p><em><strong>Keywords</strong></em>: <code class="language-plaintext highlighter-rouge">software</code>, <code class="language-plaintext highlighter-rouge">environment</code></p>

<ul>
  <li>Containerization and orchestration technologies, like Docker and Kubernetes, offer exceptional flexibility but also introduce security concerns.</li>
  <li>Securing containers and managing their lifecycle is vital to ensure a safe cloud environment.</li>
  <li>For example, use container scanning tools to identify vulnerabilities within container images before deploying them.</li>
  <li>Additionally, enforce strict security policies and segregate workloads using Kubernetes namespaces.</li>
</ul>

<h3 id="continuous-monitoring-and-incident-response">Continuous Monitoring and Incident Response</h3>
<p><em><strong>Keywords</strong></em>: <code class="language-plaintext highlighter-rouge">software</code>, <code class="language-plaintext highlighter-rouge">environment</code></p>

<ul>
  <li>The cloud landscape is constantly changing, and threats evolve rapidly. This means that we need to monitor not only for known threats but also for anomalies.</li>
  <li>Continuous monitoring and proactive incident response are essential to detect anomalies and respond swiftly to security incidents.</li>
  <li>For example, use cloud-native monitoring tools your Cloud or platform provider provides.</li>
  <li>Have good logging, but remember that more is not always better &#8211; log relevant information.</li>
</ul>

<h3 id="human-factors-including-social-engineering-misconfigurations-and-human-errors">Human Factors (including Social Engineering, Misconfigurations, and Human Errors)</h3>
<p><em><strong>Keywords</strong></em>: <code class="language-plaintext highlighter-rouge">people</code>, <code class="language-plaintext highlighter-rouge">human factors</code></p>

<ul>
  <li>82% of incidents are caused by human factors (<a href="https://www.verizon.com/business/resources/T39a/reports/dbir/2022-data-breach-investigations-report-dbir.pdf">2022 Data Breach Investigations Report</a>)</li>
  <li>Creating secure applications also implies providing security training for the system users.</li>
  <li>Social engineering and human factor has proven to be essential to creating secure applications.</li>
  <li>Consider running security awareness campaigns and employee training from user and developer perspectives.</li>
  <li>Automate routine and mundane tasks &#8211; humans often don&#8217;t enjoy carrying out tasks like this and are prone to errors; computers, on the other hand, excel at tasks like this!</li>
</ul>

<h2 id="conclusion">Conclusion</h2>
<p>You have probably heard that nothing is stronger than its weakest link. Therefore, it is important to look at various sides of the security. Especially in the Cloud, one size does not fit all when it comes to security. Cloud platforms, software, and threats constantly evolve and add to the complexity of creating secure applications.</p>

<p>Here, we have seen some of the principles to consider regarding the security of the platforms and application development for the Cloud and cloud-native applications in general.</p>

<p>Finally, note that this is not an exhaustive list but is instead meant to serve as a stepping stone to more secure application development.</p>

<hr />]]></content><author><name>Rustam Mehmandarov</name></author><summary type="html">This is the second part of the series on the Cloud Security Principles. This post will look at some key principles for securing your applications. Similarly to the first post, some prior knowledge of various IT architecture and security concepts may be expected. This post was inspired by a talk I...</summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://mehmandarov.com/assets/images/posts-images/container-ship.jpeg"/><category term="blog"/><category term="english"/><category term="architecture"/><category term="security"/><category term="cloud"/></entry><entry><title type="html">Cloud Security Principles</title><link href="https://mehmandarov.com/cloud-security-principles/" rel="alternate" type="text/html" title="Cloud Security Principles"/><published>2023-10-12T09:50:00+02:00</published><updated>2023-10-12T09:50:00+02:00</updated><id>https://mehmandarov.com/cloud-security-principles</id><content type="html" xml:base="https://mehmandarov.com/cloud-security-principles/"><![CDATA[<p><em>This post was inspired by a talk I have recently done with <a href="https://linktr.ee/nehasardana">Neha Sardana</a> at JAX New York and is meant to serve as a stepping stone to categorize and catalog the things you need to consider working with the Cloud and Cloud-native applications. Some prior knowledge of various concepts within IT architecture and security may be expected for this post.</em></p>

<ul>
  <li><a href="#introduction">Introduction</a></li>
  <li><a href="#key-elements-of-a-cloud-security-architecture">Key Elements of a Cloud Security Architecture</a></li>
  <li><a href="#responsibilities">Responsibilities</a></li>
  <li><a href="#constantly-evolving-landscape">Constantly Evolving Landscape</a></li>
  <li><a href="#platform-security-architecture">Platform Security Architecture</a></li>
  <li><a href="#application-security-architecture">Application Security Architecture</a></li>
  <li><a href="#conclusion">Conclusion</a></li>
</ul>

<hr />

<h2 id="introduction">Introduction</h2>
<p>Whether you are running on the Cloud or not it is all about the <a href="https://www.techtarget.com/whatis/definition/Confidentiality-integrity-and-availability-CIA">CIA triad model</a> &#8211; Confidentiality, Integrity, and Availability.</p>

<p>When thinking about Cloud Security Architecture we need to be able to think about the whole stack. Of course, we don&#8217;t need to think about all the moving parts alone &#8211; it is a shared responsibility between the Cloud service provider and you, the user of the platform.</p>

<h2 id="key-elements-of-a-cloud-security-architecture">Key Elements of a Cloud Security Architecture</h2>
<p>Let&#8217;s first start by defining the key elements of a Cloud Security Architecture, divided across the layers of the stack, based on the Cloud Security Alliance (CSA) stack model.</p>

<p><img src="/assets/images/posts-images/2023-10-12-fig1.png" alt="Fig.1: Cloud Security Alliance (CSA) stack model" /></p>
<figcaption class="caption">Fig.1: Cloud Security Alliance (CSA) stack model</figcaption>

<p>Now, we can also mention some of the main challenges related to security, divided into separate groups, and try to map them to the CIA triad model that we have mentioned earlier.</p>

<h3 id="network-and-storage">Network and Storage</h3>
<ul>
  <li>Data Encryption</li>
  <li>Network Security</li>
</ul>

<h3 id="application-layer">Application layer</h3>
<ul>
  <li>Application Security</li>
  <li>Logging and Monitoring</li>
  <li>Identity and Access Management (IAM)</li>
</ul>

<h3 id="observability-and-traceability">Observability, and traceability</h3>
<ul>
  <li>Incident Response and Recovery</li>
  <li>Vendor and Third-Party Risk Management</li>
</ul>

<h3 id="devops">DevOps</h3>
<ul>
  <li>Automation and Orchestration</li>
  <li>Resilience and High Availability</li>
</ul>

<h3 id="general">General</h3>
<ul>
  <li>Compliance and Governance</li>
  <li>User Training and Awareness</li>
  <li>Cloud Provider Security Features</li>
</ul>

<p><img src="/assets/images/posts-images/2023-10-12-fig2.png" alt="Fig.2: Challenges of Cloud Security" /></p>
<figcaption class="caption">Fig.2: Challenges of Cloud Security</figcaption>

<h2 id="responsibilities">Responsibilities</h2>
<h3 id="shared-responsibility--intersection-of-responsibilities">Shared Responsibility + Intersection of Responsibilities</h3>
<p>Addressing all these challenges is a shared responsibility between the Cloud service provider and the customer and the division will vary depending on the type of the solution and whether you are using IaaS, PaaS, or SaaS.</p>

<p>Typically, Cloud service providers will take care of the lower parts of the stack, like physical, infrastructure, and platform security, while customers will be responsible for creating secure applications, securing their data, creating proper Identity and Access Management (IAM), and configuration management.</p>

<p>An effective overlap and a clear understanding of the responsibilities ensure comprehensive security coverage across all layers.</p>

<p><img src="/assets/images/posts-images/2023-10-12-fig3.png" alt="Fig.3: Shared security responsibility between the Cloud Service providers and the Customers" /></p>
<figcaption class="caption">Fig.3: Shared security responsibility between the Cloud Service providers and the Customers</figcaption>

<h2 id="constantly-evolving-landscape">Constantly Evolving Landscape</h2>
<h3 id="evolving-landscape--constant-change">Evolving Landscape == Constant Change</h3>
<p>One of the differentiating factors from regular application development is the constant change and evolution of the platform and tooling on one side, and the constantly evolving types of attacks and possibly larger attack surfaces on the other side.</p>

<p>These factors will lead to changes in the model and the responsibility division. The same might be influenced by the new services being introduced both from the side of the Cloud service provider and the customer (app developer).</p>

<p>Therefore, regular communication between the parties involved and staying updated on their security practices is essential to ensure secure Cloud applications.</p>

<h2 id="types-of-the-cloud-security-architecture">Types of the Cloud Security Architecture</h2>
<p>The Cloud Security Architecture is <em>twofold</em> &#8211; you will need to choose a <em>platform</em> for running your application and think about the security of the <em>application</em> you will be deploying on that platform.</p>

<h3 id="platform-security-architecture">Platform Security Architecture</h3>
<p>Let&#8217;s start with defining the types of platforms and list some of the key elements to consider when choosing a platform type.</p>

<h4 id="public-cloud-security-architecture">Public Cloud Security Architecture</h4>
<ul>
  <li>Designed for cloud services provided by third-party vendors (e.g., AWS, Azure, Google Cloud).</li>
  <li>Focuses on securing data and applications hosted on shared infrastructure.</li>
  <li>Utilizes the security features provided by the cloud service provider (CSP) while also implementing * additional security measures.</li>
  <li>Emphasizes network segmentation, encryption, IAM, and monitoring.</li>
</ul>

<h4 id="private-cloud-security-architecture">Private Cloud Security Architecture</h4>
<ul>
  <li>Created for cloud environments dedicated to a single organization.</li>
  <li>Offers more control over security settings and configurations.</li>
  <li>Often used by organizations with strict compliance requirements or sensitive data.</li>
  <li>Implements strong access controls, encryption, and strict network isolation.</li>
</ul>

<h4 id="hybrid-cloud-security-architecture">Hybrid Cloud Security Architecture</h4>
<ul>
  <li>Combines public and private clouds to take advantage of the benefits of both deployment models</li>
  <li>Security architecture addresses integration challenges and ensures consistency across environments</li>
  <li>Emphasizes secure communication between on-premises and cloud components</li>
  <li>Requires seamless identity and access management across both environments</li>
</ul>

<h4 id="multi-cloud-security-architecture">Multi-Cloud Security Architecture</h4>
<ul>
  <li>Involves using services from multiple cloud providers simultaneously</li>
  <li>Ensures compatibility and security across diverse cloud platforms</li>
  <li>Requires careful management of authentication, authorization, data protection, and compliance measures</li>
  <li>Aims to prevent vendor lock-in and distribute risk</li>
</ul>

<h3 id="application-security-architecture">Application Security Architecture</h3>
<p>Here are some things you will need to think about when developing modern applications for the Cloud and the cloud-native world.</p>

<h4 id="1-secure-your-code">1. Secure Your Code</h4>
<ul>
  <li>Software Supply Chain Security: Securing and monitoring your artifacts and third-party libraries.</li>
  <li>Making sure the code you have written is secure: OWASP Top 10, static code analysis, coding best practices.</li>
</ul>

<h4 id="2-your-container-and-serverless-security-architecture">2. Your Container (and Serverless) Security Architecture</h4>
<ul>
  <li>Specifically addresses security for containerized applications (e.g., Containers, Kubernetes) and serverless computing (e.g., AWS Lambda, Azure Functions, Cloud Functions, or Cloud Run on Google Cloud)</li>
  <li>Focus on securing microservices, communication between them, their orchestrators, and function-as-a-service (FaaS) platforms</li>
  <li>Involves isolating containers, securing images, and managing runtime security</li>
</ul>

<h4 id="3-add-devsecops-architecture-practices">3. Add DevSecOps Architecture Practices</h4>
<ul>
  <li>Integrate security practices into the DevOps process: DevSecOps</li>
  <li>Ensure security is considered at every stage of application development and deployment</li>
  <li>Involves automated security testing, vulnerability scanning, and security policy enforcement</li>
</ul>

<h4 id="4-cross-application-and-cross-container-communication-zero-trust-security-architecture">4. Cross-application and Cross-container communication: Zero Trust Security Architecture</h4>
<ul>
  <li>Assume no trust by default and require strict authentication and authorization for all users and devices</li>
  <li>Focus on identity verification, principle of least privilege, and continuous monitoring</li>
  <li>Suitable for cloud environments where traditional perimeter defenses are less effective</li>
</ul>

<h4 id="5-physical-security-edge-cloud-security-architecture">5. Physical security: Edge Cloud Security Architecture</h4>
<ul>
  <li>Address security concerns at the edge of the network, closer to where data is generated and consumed</li>
  <li>In case of having local edge hardware devices consider also physical security of those devices</li>
  <li>Involves considerations like local processing, secure communication, and protection against threats targeting edge devices</li>
</ul>

<h4 id="6-compliance-centric-security-architecture">6. Compliance-Centric Security Architecture</h4>
<ul>
  <li>Tailored to meet specific regulatory compliance requirements (e.g., GDPR, HIPAA, PCI DSS)</li>
  <li>Focus on implementing controls and safeguards to adhere to relevant standards</li>
</ul>

<h2 id="conclusion">Conclusion</h2>
<p>We have seen the key elements of the cloud security architecture and the building blocks of the whole stack. Furthermore, we have looked at the various types and elements to consider when it comes to the security of the platforms and application development. This is a stepping stone to categorize and group some of the main things you will need to consider when working with the Cloud and cloud-native applications.</p>

<hr />
<p><em>** Illustrations in this post: Rustam Mehmandarov.</em></p>

<hr />]]></content><author><name>Rustam Mehmandarov</name></author><summary type="html">This post was inspired by a talk I have recently done with Neha Sardana at JAX New York and is meant to serve as a stepping stone to categorize and catalog the things you need to consider working with the Cloud and Cloud-native applications. Some prior knowledge of various concepts within IT arch...</summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://mehmandarov.com/assets/images/posts-images/containers.jpg"/><category term="blog"/><category term="english"/><category term="architecture"/><category term="security"/><category term="cloud"/></entry><entry><title type="html">It Is Alive! Again!</title><link href="https://mehmandarov.com/it-is-alive-again/" rel="alternate" type="text/html" title="It Is Alive! Again!"/><published>2023-01-15T18:50:00+01:00</published><updated>2023-01-15T18:50:00+01:00</updated><id>https://mehmandarov.com/it-is-alive-again</id><content type="html" xml:base="https://mehmandarov.com/it-is-alive-again/"><![CDATA[<p><em>Getting the site updated and rebooting blogging. It&#8217;s back!</em></p>

<hr />

<h2 id="welcome-back">Welcome back!</h2>

<p>Finally, after a long hibernation, this site is returning to life. Watch out for new posts coming to servers near you!</p>

<p>I look forward to seeing how the site will develop and future posts. Stay tuned!</p>

<hr />]]></content><author><name>Rustam Mehmandarov</name></author><summary type="html">Getting the site updated and rebooting blogging. It&#8217;s back!</summary><category term="blog"/><category term="english"/></entry><entry><title type="html">Building a Basic Apache Beam Pipeline in 4 Steps with Java</title><link href="https://mehmandarov.com/beam-pipeline-in-four-steps/" rel="alternate" type="text/html" title="Building a Basic Apache Beam Pipeline in 4 Steps with Java"/><published>2020-02-21T07:35:00+01:00</published><updated>2020-02-21T07:35:00+01:00</updated><id>https://mehmandarov.com/beam-pipeline-in-four-steps</id><content type="html" xml:base="https://mehmandarov.com/beam-pipeline-in-four-steps/"><![CDATA[<p><em>Getting started with building data pipelines using Apache Beam.</em></p>

<ul>
  <li><a href="#step-1-define-pipeline-options">Step 1: Define Pipeline Options</a></li>
  <li><a href="#step-2-create-the-pipeline">Step 2: Create the Pipeline</a></li>
  <li><a href="#step-3-apply-transformations">Step 3: Apply Transformations</a></li>
  <li><a href="#step-4-run-it">Step 4: Run it!</a></li>
  <li><a href="#conclusion">Conclusion</a></li>
</ul>

<hr />

<p>In this post, I would like to show you how you can get started with Apache Beam and build the first, simple data pipeline in 4 steps.</p>

<h2 id="step-1-define-pipeline-options">Step 1: Define Pipeline Options</h2>

<p>Let&#8217;s start with creating a helper object to configure our pipelines. This is not an absolute necessity, however defining the pipeline options might save you some time later, especially if your pipeline is dependent on a few arguments, that might have pre-defined, default values that you don&#8217;t want to provide at every run.</p>

<figure class="highlight"><pre><code class="language-java" data-lang="java"><span class="kd">public</span> <span class="kd">interface</span> <span class="nc">OsloCityBikeOptions</span> <span class="kd">extends</span> <span class="nc">PipelineOptions</span> <span class="o">{</span>

    <span class="cm">/**
     * By default, the code reads from a public dataset containing a subset of
     * bike station metadata for city bikes. Set this option to choose a different input file or glob
     * (i.e. partial names with *, like "*-stations.txt").
     */</span>
    <span class="nd">@Description</span><span class="o">(</span><span class="s">"Path of the file with the availability data"</span><span class="o">)</span>
    <span class="nd">@Default</span><span class="o">.</span><span class="na">String</span><span class="o">(</span><span class="s">"src/main/resources/bikedata-stations-example.txt"</span><span class="o">)</span>
    <span class="nc">String</span> <span class="nf">getStationMetadataInputFile</span><span class="o">();</span>
    <span class="kt">void</span> <span class="nf">setStationMetadataInputFile</span><span class="o">(</span><span class="nc">String</span> <span class="n">value</span><span class="o">);</span>

    <span class="c1">// some other options here...</span>
<span class="o">}</span></code></pre></figure>

<h2 id="step-2-create-the-pipeline">Step 2: Create the Pipeline</h2>

<p>Now that you have created the pipeline options object, you will need to create the pipeline object itself and provide the options to it:</p>

<figure class="highlight"><pre><code class="language-java" data-lang="java"><span class="nc">OsloCityBikeOptions</span> <span class="n">options</span> <span class="o">=</span> 
        <span class="nc">PipelineOptionsFactory</span><span class="o">.</span><span class="na">fromArgs</span><span class="o">(</span><span class="n">args</span><span class="o">)</span>
                                <span class="o">.</span><span class="na">withValidation</span><span class="o">()</span>
                                <span class="o">.</span><span class="na">as</span><span class="o">(</span><span class="nc">OsloCityBikeOptions</span><span class="o">.</span><span class="na">class</span><span class="o">);</span>

<span class="nc">Pipeline</span> <span class="n">pipeline</span> <span class="o">=</span> <span class="nc">Pipeline</span><span class="o">.</span><span class="na">create</span><span class="o">(</span><span class="n">options</span><span class="o">);</span></code></pre></figure>

<p>(<em>Check out the documentation for the <a href="https://beam.apache.org/releases/javadoc/2.19.0/org/apache/beam/sdk/options/PipelineOptionsFactory.html">PipelineOptionsFactory</a> class for the description of the methods used above.</em>)</p>

<h2 id="step-3-apply-transformations">Step 3: Apply Transformations</h2>

<p>After defining the pipeline and providing the options class, we can start by applying the transformations using <code class="language-plaintext highlighter-rouge">.apply(...)</code>. Those can be chained after each other by applying yet another <code class="language-plaintext highlighter-rouge">.apply(...)</code>, for instance:</p>

<figure class="highlight"><pre><code class="language-java" data-lang="java"><span class="nc">PCollection</span> <span class="o">&lt;</span><span class="no">KV</span><span class="o">&lt;</span><span class="nc">Integer</span><span class="o">,</span> <span class="nc">LinkedHashMap</span><span class="o">&gt;&gt;</span> <span class="n">stationMetadata</span> <span class="o">=</span> <span class="n">pipeline</span>
                <span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="s">"ReadLines: StationMetadataInputFiles"</span><span class="o">,</span> <span class="nc">TextIO</span><span class="o">.</span><span class="na">read</span><span class="o">().</span><span class="na">from</span><span class="o">(</span><span class="n">options</span><span class="o">.</span><span class="na">getStationMetadataInputFile</span><span class="o">()))</span>
                <span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="s">"Station Metadata"</span><span class="o">,</span> <span class="nc">ParDo</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">fnExtractStationMetaDataFromJSON</span><span class="o">()));</span>
                <span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="nc">MapElements</span><span class="o">.</span><span class="na">into</span><span class="o">(</span><span class="nc">TypeDescriptor</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="nc">String</span><span class="o">.</span><span class="na">class</span><span class="o">)).</span><span class="na">via</span><span class="o">(</span><span class="n">o</span> <span class="o">-&gt;</span> <span class="n">o</span><span class="o">.</span><span class="na">toString</span><span class="o">()))</span>
                <span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="s">"WriteStationMetaData"</span><span class="o">,</span> <span class="nc">TextIO</span><span class="o">.</span><span class="na">write</span><span class="o">().</span><span class="na">to</span><span class="o">(</span><span class="n">options</span><span class="o">.</span><span class="na">getMetadataOutput</span><span class="o">()));</span></code></pre></figure>

<p>Note that a <a href="https://beam.apache.org/releases/javadoc/2.19.0/org/apache/beam/sdk/values/PCollection.html"><code class="language-plaintext highlighter-rouge">PCollection&lt;T&gt;</code></a> is an immutable collection of values of type <code class="language-plaintext highlighter-rouge">T</code> and that you can provide names for the transformations as the first string argument in the <code class="language-plaintext highlighter-rouge">apply()</code>, like in the first two and the last <code class="language-plaintext highlighter-rouge">apply</code> methods.</p>

<p>Here we can also specify custom transformations that can be done in parallel. In Beam, they are being referred to as <a href="https://beam.apache.org/releases/javadoc/2.19.0/org/apache/beam/sdk/transforms/ParDo.html"><code class="language-plaintext highlighter-rouge">ParDo</code></a> methods. They are similar to the <code class="language-plaintext highlighter-rouge">Mapper</code> or <code class="language-plaintext highlighter-rouge">Reducer</code> class of a MapReduce-style algorithm. In this post, we will not be focusing on the contents of such pipeline (i.e. what it is doing), but a simple example of a <code class="language-plaintext highlighter-rouge">ParDo</code> can be looking like the second <code class="language-plaintext highlighter-rouge">apply</code> in the code above (look for the link in the <a href="#conclusion">conclusion</a> for the entire running example).</p>

<figure class="highlight"><pre><code class="language-java" data-lang="java"><span class="n">pipeline</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="s">"Station Metadata"</span><span class="o">,</span> <span class="nc">ParDo</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">fnExtractStationMetaDataFromJSON</span><span class="o">()));</span></code></pre></figure>

<h2 id="step-4-run-it">Step 4: Run it!</h2>

<p>After defining the pipeline, its options, and how they are connected, we can finally run the pipeline. The great thing about running the pipelines in Apache Beam is that it is very easy to switch between various runners. Beam provides a portable API layer for building sophisticated pipelines that may be executed across various execution engines or <em>runners</em>. In our example, we can switch from running the pipeline locally (with <a href="https://beam.apache.org/documentation/runners/direct/"><code class="language-plaintext highlighter-rouge">direct-runner</code></a>), to running the same pipeline in the Cloud as a managed service (with <a href="https://beam.apache.org/documentation/runners/dataflow/"><code class="language-plaintext highlighter-rouge">dataflow-runner</code></a>) by simply adjusting the values we provide when running the code.</p>

<h3 id="local-runner">Local runner</h3>

<p>Here is an example of running the pipeline with <code class="language-plaintext highlighter-rouge">direct-runner</code>:</p>

<figure class="highlight"><pre><code class="language-bash" data-lang="bash">mvn compile <span class="nb">exec</span>:java <span class="se">\</span>
      <span class="nt">-Pdirect-runner</span> <span class="se">\</span>
      <span class="nt">-Dexec</span>.mainClass<span class="o">=</span>com.mehmandarov.beam.OsloCityBike <span class="se">\</span>
      <span class="nt">-Dexec</span>.args<span class="o">=</span><span class="s2">"--inputFile=src/data-example.txt </span><span class="se">\</span><span class="s2">
      --output=bikedatalocal"</span></code></pre></figure>

<h3 id="dataflow-runner">Dataflow runner</h3>

<p>And here is the example of running the same pipeline in the Cloud as a managed service, using Google Cloud Dataflow. Note that most of the parameters provided are still the same, with a few additional parameters needed for this specific runner.</p>

<figure class="highlight"><pre><code class="language-bash" data-lang="bash">mvn compile <span class="nb">exec</span>:java <span class="se">\</span>
      <span class="nt">-Pdataflow-runner</span> <span class="se">\</span>
      <span class="nt">-Dexec</span>.mainClass<span class="o">=</span>com.mehmandarov.beam.OsloCityBike <span class="se">\</span>
      <span class="nt">-Dexec</span>.args<span class="o">=</span><span class="s2">"--project=rm-cx-211107 </span><span class="se">\</span><span class="s2">
      --inputFile=gs://my_oslo_bike_data/data-2018-*.txt </span><span class="se">\</span><span class="s2">
      --stagingLocation=gs://my_oslo_bike_data/testing </span><span class="se">\</span><span class="s2">
      --output=gs://my_oslo_bike_data/testing/output </span><span class="se">\</span><span class="s2">
      --tempLocation=gs://my_oslo_bike_data/testing/ </span><span class="se">\</span><span class="s2">
      --runner=DataflowRunner </span><span class="se">\</span><span class="s2">
      --region=europe-west1"</span></code></pre></figure>

<h3 id="other-runners">Other runners</h3>
<p>In case you would like to be using various runners or interested in switching between them, it might be a good idea to check the <a href="https://beam.apache.org/documentation/runners/capability-matrix/">capability matrix</a> in the documentation, as the core concepts of Beam Model can sometimes be implemented to varying degrees in each of the Beam runners.</p>

<h2 id="conclusion">Conclusion</h2>
<p>We have now seen the basic steps needed to create a simple data-parallel processing pipeline and how that can be run and deployed both in the local and managed Cloud environments. We are were also able to run the same pipeline with just a few adjustments to the command line parameters and, in our case, without any changes to the pipeline code.</p>

<p>The entire working example that we have been using here can be found in <a href="https://github.com/mehmandarov/oslocitybike-basic-beam">my GitHub repository</a>, as well as a more advanced example in <a href="https://github.com/mehmandarov/oslocitybike-beam">another repository</a>.</p>]]></content><author><name>Rustam Mehmandarov</name></author><summary type="html">Getting started with building data pipelines using Apache Beam Java SDK</summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://mehmandarov.com/assets/images/posts-images/pipes.jpg"/><category term="blog"/><category term="java"/><category term="apache beam"/><category term="data"/><category term="pipelines"/><category term="english"/></entry><entry><title type="html">Getting a Graph Representation of a Pipeline in Apache Beam</title><link href="https://mehmandarov.com/apache-beam-pipeline-graph/" rel="alternate" type="text/html" title="Getting a Graph Representation of a Pipeline in Apache Beam"/><published>2019-11-27T08:15:00+01:00</published><updated>2019-11-27T08:15:00+01:00</updated><id>https://mehmandarov.com/apache-beam-pipeline-graph</id><content type="html" xml:base="https://mehmandarov.com/apache-beam-pipeline-graph/"><![CDATA[<p><em>Getting a pipeline representation in Apache Beam explained step-by-step.</em></p>

<ul>
  <li><a href="#intro">Intro</a></li>
  <li><a href="#tldr-getting-graph-representation">TL;DR: Getting Graph Representation</a></li>
  <li><a href="#a-full-example">A Full Example</a></li>
  <li><a href="#what-now">What Now?</a></li>
</ul>

<hr />

<h2 id="intro">Intro</h2>
<p>Constructing advanced pipelines, or trying to wrap your head around the existing pipelines, in <a href="https://beam.apache.org/">Apache Beam</a> can sometimes be challenging. We have seen some nice visual representations of the pipelines in the managed Cloud versions of this software, but figuring out how to get a graph representation of the pipeline required a little bit of research. Here is how it is done in a few steps using Beam&#8217;s Java SDK.</p>

<h2 id="tldr-getting-graph-representation">TL;DR: Getting Graph Representation</h2>

<p>If you just want to see a few lines that let you generate the <a href="https://en.wikipedia.org/wiki/DOT_(graph_description_language)">DOT</a> representation of the graph, here it is:</p>

<figure class="highlight"><pre><code class="language-java" data-lang="java"><span class="kn">import</span> <span class="nn">org.apache.beam.runners.core.construction.renderer.PipelineDotRenderer</span><span class="o">;</span>

<span class="nc">Pipeline</span> <span class="n">p</span> <span class="o">=</span> <span class="nc">Pipeline</span><span class="o">.</span><span class="na">create</span><span class="o">(</span><span class="n">options</span><span class="o">);</span>
<span class="c1">// do stuff with your pipeline</span>
<span class="nc">String</span> <span class="n">dotString</span> <span class="o">=</span> <span class="nc">PipelineDotRenderer</span><span class="o">.</span><span class="na">toDotString</span><span class="o">(</span><span class="n">p</span><span class="o">);</span></code></pre></figure>

<p>Now, if you want a slightly more comprehensive example, keep on reading.</p>

<h2 id="a-full-example">A Full Example</h2>
<p>Here we will be using <a href="https://beam.apache.org/get-started/quickstart-java/#get-the-wordcount-code">word count example</a>, particularly the <a href="https://github.com/apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/MinimalWordCount.java"><code class="language-plaintext highlighter-rouge">MinimalWordCount</code></a> class.</p>

<h4 id="adding-maven-dependency">Adding Maven Dependency</h4>
<p>First, we need to add a dependency to the Maven file under <code class="language-plaintext highlighter-rouge">&lt;dependencies&gt;</code> section:</p>

<figure class="highlight"><pre><code class="language-xml" data-lang="xml"><span class="nt">&lt;dependencies&gt;</span>
    <span class="c">&lt;!-- ... all the other dependencies you may have --&gt;</span>
    <span class="nt">&lt;dependency&gt;</span>
        <span class="nt">&lt;groupId&gt;</span>org.apache.beam<span class="nt">&lt;/groupId&gt;</span>
        <span class="nt">&lt;artifactId&gt;</span>beam-runners-core-construction-java<span class="nt">&lt;/artifactId&gt;</span>
        <span class="nt">&lt;version&gt;</span>${beam.version}<span class="nt">&lt;/version&gt;</span>
    <span class="nt">&lt;/dependency&gt;</span>
<span class="nt">&lt;/dependencies&gt;</span></code></pre></figure>

<h4 id="the-code">The Code</h4>
<p>Now, we will need to add a few imports (assuming you already added the Maven dependency mentioned earlier):</p>

<figure class="highlight"><pre><code class="language-java" data-lang="java"><span class="kn">import</span> <span class="nn">org.apache.beam.runners.core.construction.renderer.PipelineDotRenderer</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">org.slf4j.Logger</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">org.slf4j.LoggerFactory</span><span class="o">;</span></code></pre></figure>

<p>To get the <a href="https://en.wikipedia.org/wiki/DOT_(graph_description_language)">DOT</a> representation of the pipeline graph we will be passing the pipeline object to the <code class="language-plaintext highlighter-rouge">PipelineDotRenderer</code> class, and in this example, we are only logging the output to the console (hence the log4j imports).</p>

<figure class="highlight"><pre><code class="language-java" data-lang="java"><span class="c1">// Create the Pipeline object with the options we defined above</span>
<span class="nc">Pipeline</span> <span class="n">p</span> <span class="o">=</span> <span class="nc">Pipeline</span><span class="o">.</span><span class="na">create</span><span class="o">(</span><span class="n">options</span><span class="o">);</span>

<span class="c1">// ... do stuff with your pipeline ...</span>

<span class="c1">// Add this piece of code just before running the pipeline:</span>
<span class="kd">final</span> <span class="nc">Logger</span> <span class="n">log</span> <span class="o">=</span> <span class="nc">LoggerFactory</span><span class="o">.</span><span class="na">getLogger</span><span class="o">(</span><span class="nc">MinimalWordCount</span><span class="o">.</span><span class="na">class</span><span class="o">);</span>
<span class="nc">String</span> <span class="n">dotString</span> <span class="o">=</span> <span class="nc">PipelineDotRenderer</span><span class="o">.</span><span class="na">toDotString</span><span class="o">(</span><span class="n">p</span><span class="o">);</span>
<span class="n">log</span><span class="o">.</span><span class="na">info</span><span class="o">(</span><span class="s">"MY GRAPH REPR: "</span> <span class="o">+</span> <span class="n">dotString</span><span class="o">);</span>

<span class="n">p</span><span class="o">.</span><span class="na">run</span><span class="o">().</span><span class="na">waitUntilFinish</span><span class="o">();</span></code></pre></figure>

<p>That&#8217;s it. To see the code in action, run it from the command line:</p>

<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span>mvn compile <span class="nb">exec</span>:java <span class="se">\</span>
        <span class="nt">-Dexec</span>.mainClass<span class="o">=</span>org.apache.beam.examples.MinimalWordCount <span class="se">\</span>
        <span class="nt">-Pdirect-runner</span></code></pre></figure>

<p>This code will produce a DOT representation of the pipeline and log it to the console.</p>

<h4 id="a-complete-example">A Complete Example</h4>

<p>A fully working example can be found in <a href="https://github.com/mehmandarov/word-count-mini-beam">my repository</a>, based on <a href="https://github.com/apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/MinimalWordCount.java"><code class="language-plaintext highlighter-rouge">MinimalWordCount</code></a>
code. There, in addition to logging to the console, we will be storing the DOT representation to a file.</p>

<p>In the next section, we will have a brief look at what can be done with the DOT representations.</p>

<h2 id="what-now">What Now?</h2>
<p>Now that we have a DOT representation of the pipeline graph, we can use it to get a better understanding of the pipeline. For instance, you can generate an SVG or a PNG image from the data. Note that the generated graph might be a bit verbose, but gives a good overview of the pipeline graph.</p>

<p>Here, I have also included examples of the <a href="https://github.com/mehmandarov/word-count-mini-beam/blob/master/pipeline_graph.dot">DOT graph</a> and the <a href="https://github.com/mehmandarov/word-count-mini-beam/blob/master/pipeline_graph.png">PNG file</a> generated for that particular pipeline.</p>

<p>Assuming that you have Graphviz <a href="https://www.graphviz.org/download/">tools</a> installed, you can convert a DOT file to a PNG image using this command:</p>

<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span>dot <span class="nt">-Tpng</span> <span class="nt">-o</span> pipeline_graph.png pipeline_graph.dot</code></pre></figure>

<p>In addition to <a href="https://www.graphviz.org/">Grapgviz</a> (Wikipedia <a href="https://en.wikipedia.org/wiki/Graphviz">link</a>), there are also online services for converting DOT graphs to graphical representations, like <a href="https://dreampuf.github.io/GraphvizOnline">this</a> one.</p>

<p><img src="https://raw.githubusercontent.com/mehmandarov/word-count-mini-beam/master/pipeline_graph_partial.png" alt="Training your own model" class="bigger-image" /></p>
<figcaption class="caption">A part of a graphical representation for the pipeline in the MinimalWordCount example. </figcaption>

<hr />]]></content><author><name>Rustam Mehmandarov</name></author><summary type="html">How to get a graph representation of your data pipeline in Apache Beam, step by step.</summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://mehmandarov.com/assets/images/posts-images/golden-gate.jpg"/><category term="blog"/><category term="java"/><category term="apache beam"/><category term="data"/><category term="pipelines"/><category term="english"/></entry><entry><title type="html">MicroProfile &#8211; Part 1: Defining End-Points</title><link href="https://mehmandarov.com/microprofile-101-part1/" rel="alternate" type="text/html" title="MicroProfile &#8211; Part 1: Defining End-Points"/><published>2019-07-29T07:25:00+02:00</published><updated>2019-07-29T07:25:00+02:00</updated><id>https://mehmandarov.com/microprofile-101-part1</id><content type="html" xml:base="https://mehmandarov.com/microprofile-101-part1/"><![CDATA[<p><em>Part 1: End-points in MicroProfile. This is a part of a series of posts to help you get started with microservices in MicroProfile and showing off some of the features it brings to the table.</em></p>

<blockquote>
  <p><strong>Update (April 2026):</strong> This post has been refreshed to use the modern <code class="language-plaintext highlighter-rouge">jakarta.*</code> namespace introduced in Jakarta EE 9+. If you are working with an older codebase that still uses the legacy <code class="language-plaintext highlighter-rouge">javax.*</code> namespace, the concepts described here apply identically &#8212; only the package prefix differs.</p>
</blockquote>

<ul>
  <li><a href="#intro">Intro</a></li>
  <li><a href="#getting-started">Getting started</a></li>
  <li><a href="#defining-end-points">Defining End-Points</a>
    <ul>
      <li><a href="#a-regular-json-end-point">A (Regular) JSON End-Point</a></li>
      <li><a href="#an-end-point-returning-an-image">An End-Point Returning an Image</a></li>
      <li><a href="#an-end-point-returning-a-pdf">An End-Point Returning a PDF</a></li>
    </ul>
  </li>
  <li><a href="#whats-next">What&#8217;s Next?</a></li>
</ul>

<hr />

<h2 id="intro">Intro</h2>

<p>In my previous two posts, I have been describing parts of a system for &#8220;checking-in&#8221; to a location using QR codes in Java. We started with <a href="/generating-qr-codes-with-secure-hashes-using-java/">generating QR codes</a>, followed by <a href="/generating-pdf-files-using-java/">generating PDF files</a>.</p>

<p>Now, I would like to focus on building microservices around that functionality. We will be creating a few HTTP end-points built with <a href="https://microprofile.io/">MicroProfile</a>. I will be using the next few posts as an opportunity to show off some of the features that you will be getting out of the box or with minimal effort using MicroProfile.</p>

<h2 id="getting-started">Getting started</h2>

<p><em>(Assuming that you have Git, Java 9+, and Maven installed.)</em></p>

<p>Since we will continue using the QR code generator project to showcase various features of MicroProfile, it might be a good idea to familiarize yourself with the code. You might want to start with taking a look at my previous two posts that explain the code for <a href="/generating-qr-codes-with-secure-hashes-using-java/">generating QR codes</a>, and <a href="/generating-pdf-files-using-java/">generating PDF files</a> in detail.</p>

<p>Now you can clone the <a href="https://github.com/mehmandarov/microprofile-qrcodes/">project</a> and examine the Maven dependencies in the <a href="https://github.com/mehmandarov/microprofile-qrcodes/blob/master/pom.xml"><code class="language-plaintext highlighter-rouge">pom.xml</code></a> file, as well as any other MicroProfile related dependencies.</p>

<p>After cloning and opening the project in your favorite IDE, build it (again, assuming that you have Java and Maven installed) with the following command in a terminal:</p>

<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span>mvn clean package</code></pre></figure>

<p>When the project is done building and you have got a <code class="language-plaintext highlighter-rouge">Build Success</code> from Maven, you can run the project to make sure everything runs fine:</p>

<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span>java <span class="nt">-jar</span> target/qrcreator.jar</code></pre></figure>

<p>In a few seconds it takes for the app server to start-up, you should be able to access the starter page at <a href="http://localhost:8181/qrcreator/index.html"><code class="language-plaintext highlighter-rouge">http://localhost:8181/qrcreator/index.html</code></a>.</p>

<h2 id="defining-end-points">Defining End-Points</h2>

<p>One of the most obvious features any microservice needs is an end-point to receive requests and respond with some kind of data back. Let&#8217;s have a closer look into how this is done in MicroProfile. First of all, we will need to define the application path that serves as the base URI for all resource URIs (think of it as a &#8220;root&#8221; URL) and make sure that the class where it is defined extends <code class="language-plaintext highlighter-rouge">jakarta.ws.rs.core.Application</code>, like in <a href="https://github.com/mehmandarov/microprofile-qrcodes/blob/master/src/main/java/com/mehmandarov/qrcreator/ApplicationEntryPoint.java"><code class="language-plaintext highlighter-rouge">ApplicationEntryPoint</code></a> class here:</p>

<figure class="highlight"><pre><code class="language-java" data-lang="java"><span class="kn">package</span> <span class="nn">com.mehmandarov.qrcreator</span><span class="o">;</span>

<span class="kn">import</span> <span class="nn">jakarta.ws.rs.ApplicationPath</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">jakarta.ws.rs.core.Application</span><span class="o">;</span>

<span class="cm">/**
 * Sets the application path that serves as the base URI for
 * all resource URIs provided by @Path annotation.
 */</span>
<span class="nd">@ApplicationPath</span><span class="o">(</span><span class="s">"/api"</span><span class="o">)</span>

<span class="kd">public</span> <span class="kd">class</span> <span class="nc">ApplicationEntryPoint</span> <span class="kd">extends</span> <span class="nc">Application</span> <span class="o">{</span>
<span class="o">}</span></code></pre></figure>

<p>This will set up all the end-point URLs to start with <code class="language-plaintext highlighter-rouge">/api</code>, in our case <a href="http://localhost:8181/qrcreator/api/"><code class="language-plaintext highlighter-rouge">http://localhost:8181/qrcreator/api/</code></a>.</p>

<h4 id="a-regular-json-end-point">A (Regular) JSON End-Point</h4>

<p>Now, let&#8217;s define some endpoints. We will start with a most regular kind &#8211; a JSON end-point. This is probably the most common end-point you will encounter.</p>

<p>We will start with creating one that would respond to requests sent to <a href="http://localhost:8181/qrcreator/api/somestring/json"><code class="language-plaintext highlighter-rouge">/api/somestring/json</code></a>. Note that as you can see from the code below, <code class="language-plaintext highlighter-rouge">@Path</code> defines <code class="language-plaintext highlighter-rouge">somestring</code> in the URL as an <code class="language-plaintext highlighter-rouge">id</code> that is passed on as an argument to the <code class="language-plaintext highlighter-rouge">createIdKeyTuple</code> method.</p>

<p>We will also define a type of a request (in this case it is a <code class="language-plaintext highlighter-rouge">GET</code> request) and specify that it will be returning a JSON document with <code class="language-plaintext highlighter-rouge">@Produces</code> annotation.</p>

<figure class="highlight"><pre><code class="language-java" data-lang="java"><span class="nd">@GET</span>
<span class="nd">@Path</span><span class="o">(</span><span class="s">"{id}/json"</span><span class="o">)</span>
<span class="nd">@Produces</span><span class="o">(</span><span class="s">"application/json"</span><span class="o">)</span>
<span class="kd">public</span> <span class="nc">Response</span> <span class="nf">createIdKeyTuple</span><span class="o">(</span><span class="nd">@PathParam</span><span class="o">(</span><span class="s">"id"</span><span class="o">)</span> <span class="nc">String</span> <span class="n">id</span><span class="o">)</span> <span class="o">{</span>
    <span class="o">...</span>
<span class="o">}</span></code></pre></figure>

<p>Now, that we are done with annotations for the method, let&#8217;s have a look at the code for this method, that defines a JSON end-point in the <a href="https://github.com/mehmandarov/microprofile-qrcodes/blob/master/src/main/java/com/mehmandarov/qrcreator/QRController.java"><code class="language-plaintext highlighter-rouge">QRController</code></a> class:</p>

<figure class="highlight"><pre><code class="language-java" data-lang="java"><span class="nd">@GET</span>
<span class="nd">@Path</span><span class="o">(</span><span class="s">"{id}/json"</span><span class="o">)</span>
<span class="nd">@Produces</span><span class="o">(</span><span class="s">"application/json"</span><span class="o">)</span>
<span class="kd">public</span> <span class="nc">Response</span> <span class="nf">createIdKeyTuple</span><span class="o">(</span><span class="nd">@PathParam</span><span class="o">(</span><span class="s">"id"</span><span class="o">)</span> <span class="nc">String</span> <span class="n">id</span><span class="o">)</span> <span class="o">{</span>
    <span class="nc">String</span> <span class="n">nameKeyTuple</span> <span class="o">=</span> <span class="kc">null</span><span class="o">;</span>
    <span class="k">try</span> <span class="o">{</span>
        <span class="n">nameKeyTuple</span> <span class="o">=</span> <span class="n">qrCodeContentsSupplier</span><span class="o">.</span><span class="na">getQRCodeContents</span><span class="o">(</span><span class="n">id</span><span class="o">);</span>
        <span class="k">return</span> <span class="nc">Response</span><span class="o">.</span><span class="na">ok</span><span class="o">(</span><span class="n">nameKeyTuple</span><span class="o">).</span><span class="na">build</span><span class="o">();</span>
    <span class="o">}</span> <span class="k">catch</span> <span class="o">(</span><span class="nc">InvalidKeySpecException</span> <span class="n">e</span><span class="o">)</span> <span class="o">{</span>
        <span class="n">e</span><span class="o">.</span><span class="na">printStackTrace</span><span class="o">();</span>
    <span class="o">}</span> <span class="k">catch</span> <span class="o">(</span><span class="nc">NoSuchAlgorithmException</span> <span class="n">e</span><span class="o">)</span> <span class="o">{</span>
        <span class="n">e</span><span class="o">.</span><span class="na">printStackTrace</span><span class="o">();</span>
    <span class="o">}</span> <span class="k">catch</span> <span class="o">(</span><span class="nc">Exception</span> <span class="n">e</span><span class="o">)</span> <span class="o">{</span>
        <span class="n">e</span><span class="o">.</span><span class="na">printStackTrace</span><span class="o">();</span>
    <span class="o">}</span>
    <span class="k">return</span> <span class="nc">Response</span><span class="o">.</span><span class="na">serverError</span><span class="o">().</span><span class="na">build</span><span class="o">();</span>
<span class="o">}</span></code></pre></figure>

<p>Note that in this code we have only simple exception handling that makes sure we return a correct HTTP status code &#8211; OK (200) on a success and server error (500) on an internal error. You can later add other HTTP codes based on your needs.</p>

<h4 id="an-end-point-returning-an-image">An End-Point Returning an Image</h4>

<p>Returning an image instead of a JSON document is quite similar to the code we have already seen. Here we will have to pay attention to three aspects:</p>

<ul>
  <li>Different MIME type defined in <code class="language-plaintext highlighter-rouge">@Produces</code> annotation: <code class="language-plaintext highlighter-rouge">@Produces("image/png")</code></li>
  <li>Additional elements in the response header that let you control how the created file is displayed in the browser, i.e. shown in the browser &#8220;inline&#8221;, or made available through a download dialog &#8211; <code class="language-plaintext highlighter-rouge">"Content-Disposition", "inline;"</code> (see specs for <a href="https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Disposition"><code class="language-plaintext highlighter-rouge">Content-Disposition</code></a> for more details)</li>
  <li>Additional elements in the response header that let you control the name for the created file: <code class="language-plaintext highlighter-rouge">filename=\"" + id + ".png\"</code></li>
</ul>

<p>Let&#8217;s have a look at the whole method:</p>

<figure class="highlight"><pre><code class="language-java" data-lang="java"><span class="nd">@GET</span>
<span class="nd">@Path</span><span class="o">(</span><span class="s">"{id}"</span><span class="o">)</span>
<span class="nd">@Produces</span><span class="o">(</span><span class="s">"image/png"</span><span class="o">)</span>
<span class="kd">public</span> <span class="nc">Response</span> <span class="nf">createQR</span><span class="o">(</span><span class="nd">@PathParam</span><span class="o">(</span><span class="s">"id"</span><span class="o">)</span> <span class="nc">String</span> <span class="n">id</span><span class="o">)</span> <span class="o">{</span>
    <span class="k">try</span> <span class="o">{</span>
        <span class="kt">byte</span><span class="o">[]</span> <span class="n">imageData</span> <span class="o">=</span> <span class="n">qrCodeSupplier</span><span class="o">.</span><span class="na">qrCodeGenerator</span><span class="o">(</span><span class="n">id</span><span class="o">);</span>
        <span class="k">return</span> <span class="nc">Response</span><span class="o">.</span><span class="na">ok</span><span class="o">(</span><span class="n">imageData</span><span class="o">)</span>
            <span class="o">.</span><span class="na">header</span><span class="o">(</span><span class="s">"Content-Disposition"</span><span class="o">,</span> <span class="s">"inline; filename=\""</span> <span class="o">+</span> <span class="n">id</span> <span class="o">+</span> <span class="s">".png\""</span><span class="o">)</span>
            <span class="o">.</span><span class="na">build</span><span class="o">();</span>
    <span class="o">}</span> <span class="k">catch</span> <span class="o">(</span><span class="nc">WriterException</span> <span class="n">e</span><span class="o">)</span> <span class="o">{</span>
        <span class="n">e</span><span class="o">.</span><span class="na">printStackTrace</span><span class="o">();</span>
    <span class="o">}</span> <span class="k">catch</span> <span class="o">(</span><span class="nc">IOException</span> <span class="n">e</span><span class="o">)</span> <span class="o">{</span>
        <span class="n">e</span><span class="o">.</span><span class="na">printStackTrace</span><span class="o">();</span>
    <span class="o">}</span> <span class="k">catch</span> <span class="o">(</span><span class="nc">NoSuchAlgorithmException</span> <span class="n">e</span><span class="o">)</span> <span class="o">{</span>
        <span class="n">e</span><span class="o">.</span><span class="na">printStackTrace</span><span class="o">();</span>
    <span class="o">}</span> <span class="k">catch</span> <span class="o">(</span><span class="nc">InvalidKeySpecException</span> <span class="n">e</span><span class="o">)</span> <span class="o">{</span>
        <span class="n">e</span><span class="o">.</span><span class="na">printStackTrace</span><span class="o">();</span>
    <span class="o">}</span> <span class="k">catch</span> <span class="o">(</span><span class="nc">Exception</span> <span class="n">e</span><span class="o">)</span> <span class="o">{</span>
        <span class="n">e</span><span class="o">.</span><span class="na">printStackTrace</span><span class="o">();</span>
    <span class="o">}</span>
    <span class="k">return</span> <span class="nc">Response</span><span class="o">.</span><span class="na">serverError</span><span class="o">().</span><span class="na">build</span><span class="o">();</span>
<span class="o">}</span></code></pre></figure>

<p>It is worth noting that the number of <code class="language-plaintext highlighter-rouge">catch</code> statements in the <code class="language-plaintext highlighter-rouge">try...catch</code> clause will vary and depend on the number and type of exceptions that can be thrown by the underlying methods.</p>

<h4 id="an-end-point-returning-a-pdf">An End-Point Returning a PDF</h4>

<p>The last method for today &#8211; defining an end-point for returning PDF files &#8211; is nearly identical to the one we used for returning images, except for one thing:</p>

<ul>
  <li>Different MIME type and explicit file encoding in <code class="language-plaintext highlighter-rouge">@Produces</code>: <code class="language-plaintext highlighter-rouge">@Produces("application/pdf; charset=utf-8")</code>.</li>
</ul>

<figure class="highlight"><pre><code class="language-java" data-lang="java"><span class="nd">@GET</span>
<span class="nd">@Path</span><span class="o">(</span><span class="s">"{id}/pdf"</span><span class="o">)</span>
<span class="nd">@Produces</span><span class="o">(</span><span class="s">"application/pdf; charset=utf-8"</span><span class="o">)</span>
<span class="kd">public</span> <span class="nc">Response</span> <span class="nf">createQRPDF</span><span class="o">(</span><span class="nd">@PathParam</span><span class="o">(</span><span class="s">"id"</span><span class="o">)</span> <span class="nc">String</span> <span class="n">id</span><span class="o">)</span> <span class="o">{</span>
    <span class="k">try</span> <span class="o">{</span>
        <span class="kt">byte</span><span class="o">[]</span> <span class="n">pdfDocument</span> <span class="o">=</span> <span class="n">pdfDocumentSupplier</span><span class="o">.</span><span class="na">pdfDocumentGenerator</span><span class="o">(</span><span class="n">id</span><span class="o">);</span>
        <span class="k">return</span> <span class="nc">Response</span><span class="o">.</span><span class="na">ok</span><span class="o">(</span><span class="n">pdfDocument</span><span class="o">)</span>
            <span class="o">.</span><span class="na">header</span><span class="o">(</span><span class="s">"Content-Disposition"</span><span class="o">,</span> <span class="s">"inline; filename=\""</span> <span class="o">+</span> <span class="n">id</span> <span class="o">+</span> <span class="s">".pdf\""</span><span class="o">)</span>
            <span class="o">.</span><span class="na">build</span><span class="o">();</span>
    <span class="o">}</span> <span class="k">catch</span> <span class="o">(</span><span class="nc">WriterException</span> <span class="n">e</span><span class="o">)</span> <span class="o">{</span>
        <span class="n">e</span><span class="o">.</span><span class="na">printStackTrace</span><span class="o">();</span>
    <span class="o">}</span> <span class="k">catch</span> <span class="o">(</span><span class="nc">IOException</span> <span class="n">e</span><span class="o">)</span> <span class="o">{</span>
        <span class="n">e</span><span class="o">.</span><span class="na">printStackTrace</span><span class="o">();</span>
    <span class="o">}</span> <span class="k">catch</span> <span class="o">(</span><span class="nc">NoSuchAlgorithmException</span> <span class="n">e</span><span class="o">)</span> <span class="o">{</span>
        <span class="n">e</span><span class="o">.</span><span class="na">printStackTrace</span><span class="o">();</span>
    <span class="o">}</span> <span class="k">catch</span> <span class="o">(</span><span class="nc">InvalidKeySpecException</span> <span class="n">e</span><span class="o">)</span> <span class="o">{</span>
        <span class="n">e</span><span class="o">.</span><span class="na">printStackTrace</span><span class="o">();</span>
    <span class="o">}</span> <span class="k">catch</span> <span class="o">(</span><span class="nc">Exception</span> <span class="n">e</span><span class="o">)</span> <span class="o">{</span>
        <span class="n">e</span><span class="o">.</span><span class="na">printStackTrace</span><span class="o">();</span>
    <span class="o">}</span>
    <span class="k">return</span> <span class="nc">Response</span><span class="o">.</span><span class="na">serverError</span><span class="o">().</span><span class="na">build</span><span class="o">();</span>
<span class="o">}</span></code></pre></figure>

<p>Of course, there are also obvious differences in the contents of the <code class="language-plaintext highlighter-rouge">byte[]</code> array, but I consider that being outside the scope of this post &#8211; you can study those differences on your own.</p>

<h2 id="whats-next">What&#8217;s Next?</h2>

<p>Here we have seen how easy it is to define an end-point that can return documents with various MIME types. In the next posts, we will be taking a closer look at things like how you can equip your end-points with metrics, provide auto-generated documentation based on OpenAPI, and add more resilience with fail-over and circuit-breakers.</p>

<hr />]]></content><author><name>Rustam Mehmandarov</name></author><summary type="html">Getting started with MicroProfile &#8211; defining end-points for JSON, images, and PDF responses.</summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://mehmandarov.com/assets/images/posts-images/lego-record-shop.jpg"/><category term="blog"/><category term="java"/><category term="microservices"/><category term="microprofile"/><category term="jakarta ee"/><category term="english"/></entry><entry><title type="html">Generating PDF Files Using Java</title><link href="https://mehmandarov.com/generating-pdf-files-using-java/" rel="alternate" type="text/html" title="Generating PDF Files Using Java"/><published>2019-06-21T07:01:00+02:00</published><updated>2019-06-21T07:01:00+02:00</updated><id>https://mehmandarov.com/generating-pdf-files-using-java</id><content type="html" xml:base="https://mehmandarov.com/generating-pdf-files-using-java/"><![CDATA[<p><em>A step by step tutorial on how to generate PDF files in Java.</em></p>

<ul>
  <li><a href="#intro">Intro</a></li>
  <li><a href="#choosing-a-library">Choosing a Library</a></li>
  <li><a href="#generating-pdf-files">Generaring PDF Files</a>
    <ul>
      <li><a href="#adding-maven-dependency">Adding Maven Dependency</a></li>
      <li><a href="#getting-started-with-the-code">Getting Started With the Code</a></li>
      <li><a href="#adding-text">Adding Text</a></li>
      <li><a href="#adding-images">Adding Images</a></li>
      <li><a href="#adding-metadata-to-the-document">Adding Metadata to the Document</a></li>
      <li><a href="#closing-saving-and-returning-the-document">Closing, Saving and Returning the Document</a></li>
    </ul>
  </li>
  <li><a href="#what-now">What Now?</a></li>
</ul>

<hr />

<h2 id="intro">Intro</h2>

<p>In my previous post, I have started describing a <a href="/generating-qr-codes-with-secure-hashes-using-java/">system for &#8220;checking-in&#8221; to a location using QR codes in Java</a>. Now, I would like to describe another part of that system that will show you how to get started with generating PDF files using Java.</p>

<p>So, let&#8217;s have a look at how this can be implemented in your solution &#8211; step by step.</p>

<h2 id="choosing-a-library">Choosing a Library</h2>

<p>Generating PDFs is normally something you would like to do using 3rd party libraries, and there are quite a few alternatives available. While choosing a library it might be a good idea to have a closer look at the licensing for that library &#8211; some of them might be very permissive and some might force your code to comply to a specific licensing. Some libraries even come with dual licensing, one under a proprietary model and one supporting an open source model.</p>

<p>One should also consider other aspects like maturity and whether it is a high- or low-level library. The latter will tell you how much code you will actually end up writing to implement your features. In the course of this project, I ended up trying two different libraries &#8211; <a href="https://itextpdf.com/en">iText</a> and, later, <a href="https://pdfbox.apache.org/">Apache PDFBox</a>. Since the point of this code was a tutorial, I decided to stick with PDFBox as it is distributed under more permissive license &#8211;&#160;<a href="https://tldrlegal.com/license/apache-license-2.0-(apache-2.0)">Apache License 2.0</a>, as opposed to dual licensed iText that is under <a href="https://tldrlegal.com/license/gnu-affero-general-public-license-v3-(agpl-3.0)">AGPL</a> and a commercial license.</p>

<h2 id="generating-pdf-files">Generating PDF Files</h2>

<p>As mentioned earlier, this library provides quite extensive functionality for generating PDF files, but it is also quite low-level, so you will have to be prepared to implement a few things you might usually take for granted, e.g. things like calculating coordinates for text that has to be centered on a page and a few other things. However, the library has a great community, so it is quite easy to get help.</p>

<h4 id="adding-maven-dependency">Adding Maven Dependency</h4>
<p>Ok, let&#8217;s get started. First things first, you will need to add the following dependencies to your <code class="language-plaintext highlighter-rouge">pom.xml</code> to use PDFBox (assuming you are using Maven to build your project):</p>

<figure class="highlight"><pre><code class="language-xml" data-lang="xml">    <span class="nt">&lt;dependency&gt;</span>
      <span class="nt">&lt;groupId&gt;</span>org.apache.pdfbox<span class="nt">&lt;/groupId&gt;</span>
      <span class="nt">&lt;artifactId&gt;</span>pdfbox<span class="nt">&lt;/artifactId&gt;</span>
      <span class="nt">&lt;version&gt;</span>2.0.15<span class="nt">&lt;/version&gt;</span>
    <span class="nt">&lt;/dependency&gt;</span></code></pre></figure>

<h4 id="getting-started-with-the-code">Getting Started With the Code</h4>
<p>For this post, I decided to paste the source code for the whole function doing the PDF generation and separate it with a few sentences, explaining the most interesting parts of the code. You can always piece the code together, or just have a look at the code in <a href="https://github.com/mehmandarov/microprofile-qrcodes/blob/master/src/main/java/com/mehmandarov/qrcreator/document/PDFDocumentSupplier.java">my repo</a>. (<em>Bonus: If you are interested in how much work it was to port the code from iText to PDFBox, this <a href="https://github.com/mehmandarov/microprofile-qrcodes/commit/bf53ec918dbc0a2e64c1c830aa4995ef63cee52e">commit</a> should give you a rough idea.</em>)</p>

<p>We start with defining a document object, a page object and add a page to a document. Afterwards, we create a content stream object that will be added to the page and document objects. This object will be responsible for holding the text and images we will be generating here.</p>

<figure class="highlight"><pre><code class="language-java" data-lang="java"><span class="c1">// Assume that the following variables are declared and set:</span>
<span class="c1">//   QRCodeSupplier qrCodeSupplier - to generate QR codes, shown in the previous post</span>
<span class="c1">//   String id - a string that will be shown on the top of the PDF file and used in the QR code</span>
<span class="c1">//   String timeZone - a string containing current time zone</span>

<span class="nc">String</span> <span class="n">headerTitle</span> <span class="o">=</span> <span class="n">id</span><span class="o">;</span>
<span class="nc">PDFont</span> <span class="n">headerFont</span> <span class="o">=</span> <span class="nc">PDType1Font</span><span class="o">.</span><span class="na">COURIER_BOLD</span><span class="o">;</span>

<span class="kt">int</span> <span class="n">marginTop</span> <span class="o">=</span> <span class="mi">30</span><span class="o">;</span>
<span class="kt">int</span> <span class="n">fontSize</span> <span class="o">=</span> <span class="mi">30</span><span class="o">;</span>

<span class="nc">PDDocument</span> <span class="n">document</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">PDDocument</span><span class="o">();</span>

<span class="nc">PDPage</span> <span class="n">page</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">PDPage</span><span class="o">(</span><span class="nc">PDRectangle</span><span class="o">.</span><span class="na">A4</span><span class="o">);</span>
<span class="nc">PDRectangle</span> <span class="n">mediaBox</span> <span class="o">=</span> <span class="n">page</span><span class="o">.</span><span class="na">getMediaBox</span><span class="o">();</span>
<span class="n">document</span><span class="o">.</span><span class="na">addPage</span><span class="o">(</span><span class="n">page</span><span class="o">);</span>

<span class="nc">PDPageContentStream</span> <span class="n">contentStream</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">PDPageContentStream</span><span class="o">(</span><span class="n">document</span><span class="o">,</span> 
                                                            <span class="n">page</span><span class="o">,</span> 
                                                            <span class="nc">PDPageContentStream</span><span class="o">.</span><span class="na">AppendMode</span><span class="o">.</span><span class="na">APPEND</span><span class="o">,</span> 
                                                            <span class="kc">true</span><span class="o">);</span></code></pre></figure>

<h4 id="adding-text">Adding Text</h4>
<p>Now we will need to calculate the coordinates for the header text string and make sure it will appear centered independent of the font and size. This is one of the &#8220;low-level&#8221; parts you will have to deal with when using PDFBox.</p>

<figure class="highlight"><pre><code class="language-java" data-lang="java"><span class="c1">// calculate coordinates to center the header text</span>
<span class="kt">float</span> <span class="n">titleWidth</span> <span class="o">=</span> <span class="n">headerFont</span><span class="o">.</span><span class="na">getStringWidth</span><span class="o">(</span><span class="n">headerTitle</span><span class="o">)</span> <span class="o">/</span> <span class="mi">1000</span> <span class="o">*</span> <span class="n">fontSize</span><span class="o">;</span>
<span class="kt">float</span> <span class="n">titleHeight</span> <span class="o">=</span> <span class="n">headerFont</span><span class="o">.</span><span class="na">getFontDescriptor</span><span class="o">().</span><span class="na">getFontBoundingBox</span><span class="o">().</span><span class="na">getHeight</span><span class="o">()</span> <span class="o">/</span> <span class="mi">1000</span> <span class="o">*</span> <span class="n">fontSize</span><span class="o">;</span>
<span class="kt">float</span> <span class="n">titleStartX</span> <span class="o">=</span> <span class="o">(</span><span class="n">mediaBox</span><span class="o">.</span><span class="na">getWidth</span><span class="o">()</span> <span class="o">-</span> <span class="n">titleWidth</span><span class="o">)</span> <span class="o">/</span> <span class="mi">2</span><span class="o">;</span>
<span class="kt">float</span> <span class="n">titleStartY</span> <span class="o">=</span> <span class="n">mediaBox</span><span class="o">.</span><span class="na">getHeight</span><span class="o">()</span> <span class="o">-</span> <span class="n">marginTop</span> <span class="o">-</span> <span class="n">titleHeight</span><span class="o">;</span></code></pre></figure>

<p>Next, we will be adding the text itself and setting font and coordinates for it on the page:</p>

<figure class="highlight"><pre><code class="language-java" data-lang="java"><span class="c1">// add header text to the document</span>
<span class="c1">// Note: This solution will not support fixed-width paragraphs and text flow</span>
<span class="n">contentStream</span><span class="o">.</span><span class="na">beginText</span><span class="o">();</span>
<span class="n">contentStream</span><span class="o">.</span><span class="na">setFont</span><span class="o">(</span><span class="n">headerFont</span><span class="o">,</span> <span class="n">fontSize</span><span class="o">);</span>
<span class="n">contentStream</span><span class="o">.</span><span class="na">newLineAtOffset</span><span class="o">(</span><span class="n">titleStartX</span><span class="o">,</span> <span class="n">titleStartY</span><span class="o">);</span>
<span class="n">contentStream</span><span class="o">.</span><span class="na">showText</span><span class="o">(</span><span class="n">headerTitle</span><span class="o">);</span>
<span class="n">contentStream</span><span class="o">.</span><span class="na">endText</span><span class="o">();</span></code></pre></figure>

<h4 id="adding-images">Adding Images</h4>
<p>Now, let&#8217;s examine how to add an image to a PDF document. Here, you can use <code class="language-plaintext highlighter-rouge">createFromFile()</code> in case your image is already available, or <code class="language-plaintext highlighter-rouge">createFromImage()</code> is you are generating the image on the fly and/or returning it from another function.</p>

<p>Below, you will also find examples of code to scale and to calculate coordinates for centering the image:</p>

<figure class="highlight"><pre><code class="language-java" data-lang="java"><span class="c1">// get image as a byte array</span>
<span class="nc">ByteArrayInputStream</span> <span class="n">bais</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">ByteArrayInputStream</span><span class="o">(</span><span class="n">qrCodeSupplier</span><span class="o">.</span><span class="na">qrCodeGenerator</span><span class="o">(</span><span class="n">id</span><span class="o">));</span>
<span class="nc">BufferedImage</span> <span class="n">bim</span> <span class="o">=</span> <span class="nc">ImageIO</span><span class="o">.</span><span class="na">read</span><span class="o">(</span><span class="n">bais</span><span class="o">);</span>

<span class="c1">// convert image to an object that can be added to the PDF document</span>
<span class="nc">PDImageXObject</span> <span class="n">pdImage</span> <span class="o">=</span> <span class="nc">LosslessFactory</span><span class="o">.</span><span class="na">createFromImage</span><span class="o">(</span><span class="n">document</span><span class="o">,</span> <span class="n">bim</span><span class="o">);</span>

<span class="c1">// calculate coordinates to center the image</span>
<span class="kt">float</span> <span class="n">scale</span> <span class="o">=</span> <span class="mi">1</span><span class="n">f</span><span class="o">;</span>
<span class="kt">int</span> <span class="n">imageOffset</span> <span class="o">=</span> <span class="mi">100</span><span class="o">;</span>

<span class="kt">float</span> <span class="n">imageWidth</span> <span class="o">=</span> <span class="n">pdImage</span><span class="o">.</span><span class="na">getWidth</span><span class="o">()</span> <span class="o">*</span> <span class="n">scale</span><span class="o">;</span>
<span class="kt">float</span> <span class="n">imageHeight</span> <span class="o">=</span> <span class="n">pdImage</span><span class="o">.</span><span class="na">getHeight</span><span class="o">()</span> <span class="o">*</span> <span class="n">scale</span><span class="o">;</span>
<span class="kt">float</span> <span class="n">imageStartX</span> <span class="o">=</span> <span class="o">(</span><span class="n">mediaBox</span><span class="o">.</span><span class="na">getWidth</span><span class="o">()</span> <span class="o">-</span> <span class="n">imageWidth</span><span class="o">)</span> <span class="o">/</span> <span class="mi">2</span><span class="o">;</span>
<span class="kt">float</span> <span class="n">imageStartY</span> <span class="o">=</span> <span class="n">titleStartY</span> <span class="o">-</span> <span class="n">imageHeight</span> <span class="o">-</span> <span class="n">imageOffset</span><span class="o">;</span>

<span class="c1">// add image into the document</span>
<span class="n">contentStream</span><span class="o">.</span><span class="na">drawImage</span><span class="o">(</span><span class="n">pdImage</span><span class="o">,</span> <span class="n">imageStartX</span><span class="o">,</span> <span class="n">imageStartY</span><span class="o">,</span> 
                        <span class="n">pdImage</span><span class="o">.</span><span class="na">getWidth</span><span class="o">()</span> <span class="o">*</span> <span class="n">scale</span><span class="o">,</span> <span class="n">pdImage</span><span class="o">.</span><span class="na">getHeight</span><span class="o">()</span> <span class="o">*</span> <span class="n">scale</span><span class="o">);</span>

<span class="c1">// closing the stream</span>
<span class="n">contentStream</span><span class="o">.</span><span class="na">close</span><span class="o">();</span></code></pre></figure>

<h4 id="adding-metadata-to-the-document">Adding Metadata to the Document</h4>
<p>Here, you can see a few examples of how you can add metadata to your document. This is done with a help of <a href="https://pdfbox.apache.org/docs/2.0.13/javadocs/org/apache/pdfbox/pdmodel/PDDocumentInformation.html">a few methods available thought the API</a>:</p>

<figure class="highlight"><pre><code class="language-java" data-lang="java"><span class="c1">// add metadata</span>
<span class="n">document</span><span class="o">.</span><span class="na">getDocumentInformation</span><span class="o">().</span><span class="na">setTitle</span><span class="o">(</span><span class="s">"Generated QR code for "</span> <span class="o">+</span> <span class="n">id</span> <span class="o">+</span> <span class="s">"."</span><span class="o">);</span>
<span class="n">document</span><span class="o">.</span><span class="na">getDocumentInformation</span><span class="o">().</span><span class="na">setSubject</span><span class="o">(</span><span class="s">"with a secure string"</span><span class="o">);</span>
<span class="n">document</span><span class="o">.</span><span class="na">getDocumentInformation</span><span class="o">().</span><span class="na">setAuthor</span><span class="o">(</span><span class="s">"rm"</span><span class="o">);</span>
<span class="n">document</span><span class="o">.</span><span class="na">getDocumentInformation</span><span class="o">().</span><span class="na">setCreator</span><span class="o">(</span><span class="s">"rm"</span><span class="o">);</span>
<span class="n">document</span><span class="o">.</span><span class="na">getDocumentInformation</span><span class="o">().</span><span class="na">setCreationDate</span><span class="o">(</span><span class="n">date</span><span class="o">);</span></code></pre></figure>

<h4 id="closing-saving-and-returning-the-document">Closing, Saving and Returning the Document</h4>
<p>After your document is built and you have added all the contents, remember to save and close your document. Now you can either save the document to the file with <a href="https://pdfbox.apache.org/1.8/cookbook/documentcreation.html"><code class="language-plaintext highlighter-rouge">document.save()</code></a>, or returning the document as a byte array to another function with <code class="language-plaintext highlighter-rouge">byteArrayOutputStream.toByteArray()</code>:</p>

<figure class="highlight"><pre><code class="language-java" data-lang="java"><span class="c1">// save and close document</span>
<span class="nc">ByteArrayOutputStream</span> <span class="n">byteArrayOutputStream</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">ByteArrayOutputStream</span><span class="o">();</span>
<span class="n">document</span><span class="o">.</span><span class="na">save</span><span class="o">(</span><span class="n">byteArrayOutputStream</span><span class="o">);</span>
<span class="n">document</span><span class="o">.</span><span class="na">close</span><span class="o">();</span>

<span class="c1">// return document as byte[]</span>
<span class="k">return</span> <span class="n">byteArrayOutputStream</span><span class="o">.</span><span class="na">toByteArray</span><span class="o">();</span></code></pre></figure>

<h2 id="what-now">What Now?</h2>

<p>In the last two posts, we have seen how to generate QR codes with a hashed string and PDF files with Java. In the next post, I will be showing how to put it all together into a <a href="https://tldrlegal.com/license/gnu-affero-general-public-license-v3-(agpl-3.0)">MicroProfile</a> microservice.</p>

<hr />]]></content><author><name>Rustam Mehmandarov</name></author><summary type="html">A step-by-step tutorial on generating PDF files in Java with text, images, and metadata.</summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://mehmandarov.com/assets/images/posts-images/pages.jpg"/><category term="blog"/><category term="java"/><category term="pdf"/><category term="english"/></entry><entry><title type="html">Generating QR Codes With Secure Hashes Using Java</title><link href="https://mehmandarov.com/generating-qr-codes-with-secure-hashes-using-java/" rel="alternate" type="text/html" title="Generating QR Codes With Secure Hashes Using Java"/><published>2019-06-10T07:01:00+02:00</published><updated>2019-06-10T07:01:00+02:00</updated><id>https://mehmandarov.com/generating-qr-codes-with-secure-hashes-using-java</id><content type="html" xml:base="https://mehmandarov.com/generating-qr-codes-with-secure-hashes-using-java/"><![CDATA[<p><em>A step by step tutorial on how to generate QR codes and secure hashed strings with salt in Java.</em></p>

<ul>
  <li><a href="#intro">Intro</a></li>
  <li><a href="#generating-qr-codes">Generaring QR Codes</a></li>
  <li><a href="#hashing-strings">Hashing Strings</a></li>
  <li><a href="#what-now">What Now?</a></li>
</ul>

<hr />

<h2 id="intro">Intro</h2>

<p>I have been testing out new functionality for &#8220;checking-in&#8221; to a location using QR codes. To make sure the user is at the specified location and is scanning my QR code (and not a &#8220;fake&#8221; code created by someone else), I needed to add a way of &#8220;signing&#8221; each code with a value that only I &#8211; the provider of the QR code &#8211; could know. This would also make it simple enough to be able to implement the same mechanism in the app used to scan the codes to verify the validity on the client side.</p>

<p>I ended up with a solution where I would have a QR code containing a JSON object with a <code class="language-plaintext highlighter-rouge">Name</code> and a <code class="language-plaintext highlighter-rouge">Key</code> &#8211; a hashed and salted name string. The string will be read by the client app used to scan the code and hashed using the same algorithm with the same secret salt value, and compared to the value in the QR code on the client side.</p>

<p>The data structure inside a generated QR code would be like this:</p>

<figure class="highlight"><pre><code class="language-json" data-lang="json"><span class="p">{</span><span class="w"> 
    </span><span class="s2">"Name"</span><span class="p">,</span><span class="w"> </span><span class="s2">"MyString"</span><span class="p">,</span><span class="w">
    </span><span class="s2">"Key"</span><span class="p">,</span><span class="w"> </span><span class="s2">"HashedMyStringWithSecretSalt"</span><span class="w">
</span><span class="p">}</span></code></pre></figure>

<p>When it comes to the implementation, I decided to do the generation or codes in Java and, later, implement this as a standalone microservice. Here, I must admit that I was surprised by how simple it was using a specialized library. More about that below.</p>

<p>So, let&#8217;s have a look at how this can be implemented in your solution &#8211; step by step.</p>

<h2 id="generating-qr-codes">Generating QR Codes</h2>

<p>First, I needed a library that can handle QR codes, and I decided to use <a href="https://github.com/zxing/zxing">Zebra Crossing  (&#8220;ZXing&#8221;)</a> library because of its simplicity and popularity (i.e. community around it).</p>

<p>All you need to get started is to add the following dependencies to your <code class="language-plaintext highlighter-rouge">pom.xml</code> (assuming you are using Maven to build your project):</p>

<figure class="highlight"><pre><code class="language-xml" data-lang="xml"><span class="nt">&lt;dependency&gt;</span>
  <span class="nt">&lt;groupId&gt;</span>com.google.zxing<span class="nt">&lt;/groupId&gt;</span>
  <span class="nt">&lt;artifactId&gt;</span>core<span class="nt">&lt;/artifactId&gt;</span>
  <span class="nt">&lt;version&gt;</span>3.4.0<span class="nt">&lt;/version&gt;</span>
<span class="nt">&lt;/dependency&gt;</span>
<span class="nt">&lt;dependency&gt;</span>
  <span class="nt">&lt;groupId&gt;</span>com.google.zxing<span class="nt">&lt;/groupId&gt;</span>
  <span class="nt">&lt;artifactId&gt;</span>javase<span class="nt">&lt;/artifactId&gt;</span>
  <span class="nt">&lt;version&gt;</span>3.4.0<span class="nt">&lt;/version&gt;</span>
<span class="nt">&lt;/dependency&gt;</span></code></pre></figure>

<p>This library provides quite an extensive functionality both for generating and reading codes. This was more than enough for my use case where I just needed to generate a QR code with a simple JSON object:</p>

<figure class="highlight"><pre><code class="language-java" data-lang="java"><span class="kd">public</span> <span class="kt">byte</span><span class="o">[]</span> <span class="nf">qrCodeGenerator</span><span class="o">(</span><span class="nc">String</span> <span class="n">id</span><span class="o">)</span> <span class="kd">throws</span> <span class="nc">IOException</span><span class="o">,</span> 
                                                <span class="nc">WriterException</span><span class="o">,</span> 
                                                <span class="nc">InvalidKeySpecException</span><span class="o">,</span> 
                                                <span class="nc">NoSuchAlgorithmException</span> <span class="o">{</span>

    <span class="nc">String</span> <span class="n">filePath</span> <span class="o">=</span> <span class="s">"QRCode.png"</span><span class="o">;</span>
    <span class="nc">String</span> <span class="n">charset</span> <span class="o">=</span> <span class="s">"UTF-8"</span><span class="o">;</span>
    <span class="nc">Map</span> <span class="n">hintMap</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">HashMap</span><span class="o">();</span>
    <span class="n">hintMap</span><span class="o">.</span><span class="na">put</span><span class="o">(</span><span class="nc">EncodeHintType</span><span class="o">.</span><span class="na">ERROR_CORRECTION</span><span class="o">,</span> <span class="nc">ErrorCorrectionLevel</span><span class="o">.</span><span class="na">L</span><span class="o">);</span>

    <span class="nc">Map</span><span class="o">&lt;</span><span class="nc">String</span><span class="o">,</span> <span class="nc">String</span><span class="o">&gt;</span> <span class="n">qrCodeDataMap</span> <span class="o">=</span> <span class="nc">Map</span><span class="o">.</span><span class="na">of</span><span class="o">(</span>
            <span class="s">"Name"</span><span class="o">,</span> <span class="n">id</span><span class="o">,</span>
            <span class="s">"Key"</span><span class="o">,</span> <span class="n">keyProvider</span><span class="o">.</span><span class="na">generateVerificationKey</span><span class="o">(</span><span class="n">id</span><span class="o">)</span> 
            <span class="c1">// see next section for &#180;generateVerificationKey&#180; method</span>
    <span class="o">);</span>

    <span class="nc">String</span> <span class="n">jsonString</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">JSONObject</span><span class="o">(</span><span class="n">qrCodeDataMap</span><span class="o">).</span><span class="na">toString</span><span class="o">();</span>
    <span class="n">createQRCode</span><span class="o">(</span><span class="n">jsonString</span><span class="o">,</span> <span class="n">filePath</span><span class="o">,</span> <span class="n">charset</span><span class="o">,</span> <span class="n">hintMap</span><span class="o">,</span> <span class="mi">500</span><span class="o">,</span> <span class="mi">500</span><span class="o">);</span>

    <span class="nc">BufferedImage</span> <span class="n">image</span> <span class="o">=</span> <span class="nc">ImageIO</span><span class="o">.</span><span class="na">read</span><span class="o">(</span><span class="k">new</span> <span class="nc">File</span><span class="o">(</span><span class="n">filePath</span><span class="o">));</span>
    <span class="nc">ByteArrayOutputStream</span> <span class="n">baos</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">ByteArrayOutputStream</span><span class="o">();</span>
    <span class="nc">ImageIO</span><span class="o">.</span><span class="na">write</span><span class="o">(</span><span class="n">image</span><span class="o">,</span> <span class="s">"png"</span><span class="o">,</span> <span class="n">baos</span><span class="o">);</span>
    <span class="kt">byte</span><span class="o">[]</span> <span class="n">imageData</span> <span class="o">=</span> <span class="n">baos</span><span class="o">.</span><span class="na">toByteArray</span><span class="o">();</span>

    <span class="k">return</span> <span class="n">imageData</span><span class="o">;</span>
<span class="o">}</span>

<span class="kd">private</span> <span class="kt">void</span> <span class="nf">createQRCode</span><span class="o">(</span><span class="nc">String</span> <span class="n">qrCodeData</span><span class="o">,</span> 
                          <span class="nc">String</span> <span class="n">filePath</span><span class="o">,</span> 
                          <span class="nc">String</span> <span class="n">charset</span><span class="o">,</span> 
                          <span class="nc">Map</span> <span class="n">hintMap</span><span class="o">,</span> 
                          <span class="kt">int</span> <span class="n">qrCodeHeight</span><span class="o">,</span> 
                          <span class="kt">int</span> <span class="n">qrCodeWidth</span><span class="o">)</span> <span class="kd">throws</span> <span class="nc">WriterException</span><span class="o">,</span> 
                                                  <span class="nc">IOException</span> <span class="o">{</span>

    <span class="nc">BitMatrix</span> <span class="n">matrix</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">MultiFormatWriter</span><span class="o">().</span><span class="na">encode</span><span class="o">(</span>
            <span class="k">new</span> <span class="nf">String</span><span class="o">(</span><span class="n">qrCodeData</span><span class="o">.</span><span class="na">getBytes</span><span class="o">(</span><span class="n">charset</span><span class="o">),</span> <span class="n">charset</span><span class="o">),</span>
            <span class="nc">BarcodeFormat</span><span class="o">.</span><span class="na">QR_CODE</span><span class="o">,</span>
            <span class="n">qrCodeWidth</span><span class="o">,</span>
            <span class="n">qrCodeHeight</span><span class="o">,</span>
            <span class="n">hintMap</span>
    <span class="o">);</span>

    <span class="nc">MatrixToImageWriter</span><span class="o">.</span><span class="na">writeToPath</span><span class="o">(</span>
            <span class="n">matrix</span><span class="o">,</span>
            <span class="n">filePath</span><span class="o">.</span><span class="na">substring</span><span class="o">(</span><span class="n">filePath</span><span class="o">.</span><span class="na">lastIndexOf</span><span class="o">(</span><span class="sc">'.'</span><span class="o">)</span> <span class="o">+</span> <span class="mi">1</span><span class="o">),</span>
            <span class="nc">FileSystems</span><span class="o">.</span><span class="na">getDefault</span><span class="o">().</span><span class="na">getPath</span><span class="o">(</span><span class="n">filePath</span><span class="o">)</span>
    <span class="o">);</span>
<span class="o">}</span></code></pre></figure>

<p>Note also fun little thing &#8211; the conversion of Java hashmaps to a JSON object using <code class="language-plaintext highlighter-rouge">JSONObject</code>. Sometimes it is much easier to build up data structure the way you want it, and then serialize to JSON:</p>

<figure class="highlight"><pre><code class="language-java" data-lang="java"><span class="nc">Map</span><span class="o">&lt;</span><span class="nc">String</span><span class="o">,</span> <span class="nc">String</span><span class="o">&gt;</span> <span class="n">qrCodeDataMap</span> <span class="o">=</span> <span class="nc">Map</span><span class="o">.</span><span class="na">of</span><span class="o">(</span>
        <span class="s">"Name"</span><span class="o">,</span> <span class="s">"SampleText"</span><span class="o">,</span>
        <span class="s">"Key"</span><span class="o">,</span> <span class="s">"SomeHashedValue"</span>
<span class="o">);</span>
<span class="nc">String</span> <span class="n">jsonString</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">JSONObject</span><span class="o">(</span><span class="n">qrCodeDataMap</span><span class="o">).</span><span class="na">toString</span><span class="o">();</span></code></pre></figure>

<p>To be able to use <code class="language-plaintext highlighter-rouge">JSONObject</code> class, you would need to add the following dependency to your <code class="language-plaintext highlighter-rouge">pom.xml</code>:</p>

<figure class="highlight"><pre><code class="language-xml" data-lang="xml"><span class="nt">&lt;dependency&gt;</span>
  <span class="nt">&lt;groupId&gt;</span>org.json<span class="nt">&lt;/groupId&gt;</span>
  <span class="nt">&lt;artifactId&gt;</span>json<span class="nt">&lt;/artifactId&gt;</span>
  <span class="nt">&lt;version&gt;</span>20180813<span class="nt">&lt;/version&gt;</span>
<span class="nt">&lt;/dependency&gt;</span></code></pre></figure>

<p>If you are looking for a more simplified interface, you might also check out <a href="https://github.com/kenglxn/QRGen">QRGen</a> that claims to simplify QR code generation API for Java even further and is built on top ZXing. However, ZXing was absolutely fine in my case.</p>

<h2 id="hashing-strings">Hashing Strings</h2>

<p>Now, I needed to be able to hash a string in a quick and secure manner. For this, I decided to use the <a href="https://www.owasp.org/index.php/Hashing_Java">method suggested by OWASP for Java</a>. To implement this method you will need to start with updating your <code class="language-plaintext highlighter-rouge">pom.xml</code>:</p>

<figure class="highlight"><pre><code class="language-xml" data-lang="xml"><span class="nt">&lt;dependency&gt;</span>
  <span class="nt">&lt;groupId&gt;</span>commons-codec<span class="nt">&lt;/groupId&gt;</span>
  <span class="nt">&lt;artifactId&gt;</span>commons-codec<span class="nt">&lt;/artifactId&gt;</span>
  <span class="nt">&lt;version&gt;</span>1.12<span class="nt">&lt;/version&gt;</span>
<span class="nt">&lt;/dependency&gt;</span></code></pre></figure>

<p>And here is the (somewhat simplified) implmentation of the said method in Java:</p>

<figure class="highlight"><pre><code class="language-java" data-lang="java"><span class="kd">public</span> <span class="nc">String</span> <span class="nf">generateVerificationKey</span><span class="o">(</span><span class="nc">String</span> <span class="n">str</span><span class="o">)</span> <span class="kd">throws</span> <span class="nc">NoSuchAlgorithmException</span><span class="o">,</span>
                                                         <span class="nc">InvalidKeySpecException</span> <span class="o">{</span>
    <span class="kt">int</span> <span class="n">iterations</span> <span class="o">=</span> <span class="mi">10000</span><span class="o">;</span>
    <span class="kt">int</span> <span class="n">keyLength</span> <span class="o">=</span> <span class="mi">512</span><span class="o">;</span>

    <span class="kt">char</span><span class="o">[]</span> <span class="n">strChars</span> <span class="o">=</span> <span class="n">str</span><span class="o">.</span><span class="na">toCharArray</span><span class="o">();</span>
    <span class="kt">byte</span><span class="o">[]</span> <span class="n">saltBytes</span> <span class="o">=</span> <span class="n">salt</span><span class="o">.</span><span class="na">getBytes</span><span class="o">();</span>

    <span class="nc">SecretKeyFactory</span> <span class="n">skf</span> <span class="o">=</span> <span class="nc">SecretKeyFactory</span><span class="o">.</span><span class="na">getInstance</span><span class="o">(</span><span class="s">"PBKDF2WithHmacSHA512"</span><span class="o">);</span>
    <span class="nc">PBEKeySpec</span> <span class="n">spec</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">PBEKeySpec</span><span class="o">(</span><span class="n">strChars</span><span class="o">,</span> <span class="n">saltBytes</span><span class="o">,</span> <span class="n">iterations</span><span class="o">,</span> <span class="n">keyLength</span><span class="o">);</span>
    <span class="nc">SecretKey</span> <span class="n">key</span> <span class="o">=</span> <span class="n">skf</span><span class="o">.</span><span class="na">generateSecret</span><span class="o">(</span> <span class="n">spec</span> <span class="o">);</span>
    <span class="kt">byte</span><span class="o">[]</span> <span class="n">hashedBytes</span> <span class="o">=</span> <span class="n">key</span><span class="o">.</span><span class="na">getEncoded</span><span class="o">(</span> <span class="o">);</span>

    <span class="k">return</span> <span class="nc">Hex</span><span class="o">.</span><span class="na">encodeHexString</span><span class="o">(</span><span class="n">hashedBytes</span><span class="o">);</span>
<span class="o">}</span></code></pre></figure>

<h2 id="what-now">What Now?</h2>

<p>By now you should be able to generate QR codes with a hashed string. In the next post, I will be sharing code on how to embed and generate PDF files with this information with Java, followed by a post where it all will be put together into a <a href="https://microprofile.io/">MicroProfile</a> microservice. Stay tuned!</p>

<hr />]]></content><author><name>Rustam Mehmandarov</name></author><summary type="html">A step-by-step tutorial on generating QR codes and secure hashed strings with salt in Java.</summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://mehmandarov.com/assets/images/posts-images/black-and-white-java.jpg"/><category term="blog"/><category term="java"/><category term="english"/></entry><entry><title type="html">Configuring Slack Notifications for Google Cloud Build</title><link href="https://mehmandarov.com/slack-notifications-for-cloud-build/" rel="alternate" type="text/html" title="Configuring Slack Notifications for Google Cloud Build"/><published>2019-03-01T07:01:00+01:00</published><updated>2019-03-01T07:01:00+01:00</updated><id>https://mehmandarov.com/slack-notifications-for-cloud-build</id><content type="html" xml:base="https://mehmandarov.com/slack-notifications-for-cloud-build/"><![CDATA[<p><em>Adding Slack notifications for your Google Cloud Build jobs explained with code examples and screenshots.</em></p>

<ul>
  <li><a href="#intro">Intro</a></li>
  <li><a href="#1-before-you-begin">1. Before you begin</a></li>
  <li><a href="#2-create-a-cloud-function">2. Create a Cloud Function</a></li>
  <li><a href="#3-deploy-the-cloud-function">3. Deploy the Cloud Function</a></li>
</ul>

<hr />

<h2 id="intro">Intro</h2>

<p>Recently we decided to migrate our builds from Travis CI to Google Cloud Build to speed up the builds. The process was quite easy and flawless; however, we were still missing a few minor things. One of them was the notifications from Cloud Build to our <code class="language-plaintext highlighter-rouge">#ops</code> channel in Slack. This was slightly annoying because you would not know if the build was finished and the site was deployed, or if it failed for some reason.</p>

<p>Integrating with Cloud Build was a bit more different than what you are used to from integrations with Jenkins or Travis CI. Normally you would just create a webhook that would call an interface in the Slack API. In Cloud Build, on the other hand, everything is getting posted to the Pub/Sub queue built into the platform, and here you would just need to subscribe to the specific queue and listen for the events. To achieve the latter, you would need a small serverless function to listen for these events and to call the Slack API.</p>

<p><img src="/assets/images/posts-images/2019-03-01-architecture.png" alt="The Architecture" class="bigger-image" /></p>
<figcaption class="caption">The Architecture.</figcaption>
<p><br /></p>

<p>Note that here we will, technically, be using paid services on Google Cloud Platform, as both Cloud Build, Cloud Pub/Sub, and Cloud Functions are billable components. However, since all the components above provide a generous free tier, you will need to work hard to get passed the free tier with this setup.</p>

<ul>
  <li><strong>Cloud Build:</strong> Free first 120 builds-minutes per day for <code class="language-plaintext highlighter-rouge">Basic</code> machine type (<code class="language-plaintext highlighter-rouge">n1-standard-1</code>).</li>
  <li><strong>Cloud Pub/Sub:</strong> Free first 10GB per month (<a href="https://cloud.google.com/pubsub/pricing">pricing</a>).</li>
  <li><strong>Cloud Functions:</strong> Free first 2 million invocations per month (<a href="https://cloud.google.com/functions/pricing">pricing</a>).</li>
</ul>

<h2 id="1-before-you-begin">1. Before you begin</h2>

<h3 id="11-prepare-your-gcp-project">1.1 Prepare your GCP project</h3>

<p><em>I assume you have a Google Cloud Account, and that you have signed in to your account.</em></p>

<ol>
  <li>Select or create a Google Cloud Platform project, e.g. from the <a href="https://console.cloud.google.com/cloud-resource-manager">Manage resources page</a>.</li>
  <li>Make sure that <a href="https://cloud.google.com/billing/docs/how-to/modify-project">billing is enabled</a> for your Google Cloud Platform project.</li>
  <li>Enable the Cloud Functions and Cloud Pub/Sub. You can also enable the APIs using this <a href="https://console.cloud.google.com/flows/enableapi?apiid=cloudfunctions,pubsub">link</a>.</li>
  <li>Use <a href="https://cloud.google.com/shell/docs/quickstart">Cloud Shell</a> right from the browser, or you can <a href="https://cloud.google.com/sdk/docs/">Install and initialize the Cloud SDK</a> on your own machine.</li>
  <li>If you have installed the Cloud SDK, update and install gcloud components:</li>
</ol>

<figure class="highlight"><pre><code class="language-bash" data-lang="bash">gcloud components update <span class="o">&amp;&amp;</span>
gcloud components <span class="nb">install </span>alpha beta</code></pre></figure>

<h3 id="12-prepare-your-slack-app">1.2 Prepare your Slack App</h3>

<p><em>I assume you have Slack installed and that you have created and signed-in to your account.</em></p>

<p>Create a <a href="https://api.slack.com/apps?new_app=1">new Slack app</a>:</p>

<ol>
  <li>Choose the app&#8217;s name and your Slack team. Click Create.</li>
  <li>Click Incoming Webhooks.</li>
  <li>Activate incoming webhooks.</li>
  <li>Click Add New Webhook to Workspace. An authorization page opens.</li>
  <li>From the drop-down menu, select the channel to which you would like notifications sent.</li>
  <li>Click Authorize.</li>
  <li>A webhook for your Slack application has been created. Copy the webhook URL and save it for later use.</li>
</ol>

<h2 id="2-create-a-cloud-function">2. Create a Cloud Function</h2>

<p>We need to create a Cloud Storage bucket to stage your Cloud Functions files. Use <code class="language-plaintext highlighter-rouge">[STAGING_BUCKET_NAME]</code> that is a globally-unique bucket name (such as <code class="language-plaintext highlighter-rouge">[PROJECT-ID]_cloudbuilds</code>):</p>

<figure class="highlight"><pre><code class="language-bash" data-lang="bash">gsutil mb gs://[STAGING_BUCKET_NAME]</code></pre></figure>

<p>You should see the following output:</p>

<figure class="highlight"><pre><code class="language-bash" data-lang="bash">Creating gs://[PROJECT-ID]_cloudbuilds/[STAGING_BUCKET_NAME]...</code></pre></figure>

<p>Next, create a directory on your local system for the application code:</p>

<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nb">mkdir</span> ~/gcb_slack
<span class="nb">cd</span> ~/gcb_slack</code></pre></figure>

<p>Then, create the following two files in the <code class="language-plaintext highlighter-rouge">gcb_slack</code> directory.</p>

<p><strong>File 1:</strong> <code class="language-plaintext highlighter-rouge">package.json</code></p>

<figure class="highlight"><pre><code class="language-json" data-lang="json"><span class="p">{</span><span class="w">
  </span><span class="nl">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"google-container-slack"</span><span class="p">,</span><span class="w">
  </span><span class="nl">"version"</span><span class="p">:</span><span class="w"> </span><span class="s2">"0.0.1"</span><span class="p">,</span><span class="w">
  </span><span class="nl">"description"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Slack integration for Google Cloud Build, using Google Cloud Functions"</span><span class="p">,</span><span class="w">
  </span><span class="nl">"main"</span><span class="p">:</span><span class="w"> </span><span class="s2">"index.js"</span><span class="p">,</span><span class="w">
  </span><span class="nl">"dependencies"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
    </span><span class="nl">"@slack/client"</span><span class="p">:</span><span class="w"> </span><span class="s2">"4.10.0"</span><span class="w">
  </span><span class="p">}</span><span class="w">
</span><span class="p">}</span></code></pre></figure>

<p><strong>File 2:</strong> <code class="language-plaintext highlighter-rouge">index.js</code></p>

<p><em>Note: Make sure to update <code class="language-plaintext highlighter-rouge">SLACK_WEBHOOK_URL</code> in the code below.</em></p>

<figure class="highlight"><pre><code class="language-javascript" data-lang="javascript"><span class="kd">const</span> <span class="nx">IncomingWebhook</span> <span class="o">=</span> <span class="nf">require</span><span class="p">(</span><span class="dl">'</span><span class="s1">@slack/client</span><span class="dl">'</span><span class="p">).</span><span class="nx">IncomingWebhook</span><span class="p">;</span>
<span class="kd">const</span> <span class="nx">SLACK_WEBHOOK_URL</span> <span class="o">=</span> <span class="dl">"</span><span class="s2">&lt;INSERT YOUR WEBHOOK FROM STEP 1.2&gt;</span><span class="dl">"</span>

<span class="kd">const</span> <span class="nx">webhook</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">IncomingWebhook</span><span class="p">(</span><span class="nx">SLACK_WEBHOOK_URL</span><span class="p">);</span>

<span class="c1">// subscribe is the main function called by Cloud Functions.</span>
<span class="nx">module</span><span class="p">.</span><span class="nx">exports</span><span class="p">.</span><span class="nx">subscribe</span> <span class="o">=</span> <span class="p">(</span><span class="nx">event</span><span class="p">,</span> <span class="nx">callback</span><span class="p">)</span> <span class="o">=&gt;</span> <span class="p">{</span>
 <span class="kd">const</span> <span class="nx">build</span> <span class="o">=</span> <span class="nf">eventToBuild</span><span class="p">(</span><span class="nx">event</span><span class="p">.</span><span class="nx">data</span><span class="p">.</span><span class="nx">data</span><span class="p">);</span>

  <span class="c1">// Skip if the current status is not in the status list.</span>
  <span class="c1">// Add additional statues to list if you'd like:</span>
  <span class="c1">// QUEUED, WORKING, SUCCESS, FAILURE,</span>
  <span class="c1">// INTERNAL_ERROR, TIMEOUT, CANCELLED</span>
  <span class="kd">const</span> <span class="nx">status</span> <span class="o">=</span> <span class="p">[</span><span class="dl">'</span><span class="s1">SUCCESS</span><span class="dl">'</span><span class="p">,</span> <span class="dl">'</span><span class="s1">FAILURE</span><span class="dl">'</span><span class="p">,</span> <span class="dl">'</span><span class="s1">INTERNAL_ERROR</span><span class="dl">'</span><span class="p">,</span> <span class="dl">'</span><span class="s1">TIMEOUT</span><span class="dl">'</span><span class="p">];</span>
  <span class="k">if </span><span class="p">(</span><span class="nx">status</span><span class="p">.</span><span class="nf">indexOf</span><span class="p">(</span><span class="nx">build</span><span class="p">.</span><span class="nx">status</span><span class="p">)</span> <span class="o">===</span> <span class="o">-</span><span class="mi">1</span><span class="p">)</span> <span class="p">{</span>
    <span class="k">return</span> <span class="nf">callback</span><span class="p">();</span>
  <span class="p">}</span>

  <span class="c1">// Send message to Slack.</span>
  <span class="kd">const</span> <span class="nx">message</span> <span class="o">=</span> <span class="nf">createSlackMessage</span><span class="p">(</span><span class="nx">build</span><span class="p">);</span>
  <span class="nx">webhook</span><span class="p">.</span><span class="nf">send</span><span class="p">(</span><span class="nx">message</span><span class="p">,</span> <span class="nx">callback</span><span class="p">);</span>
<span class="p">};</span>

<span class="c1">// eventToBuild transforms pubsub event message to a build object.</span>
<span class="kd">const</span> <span class="nx">eventToBuild</span> <span class="o">=</span> <span class="p">(</span><span class="nx">data</span><span class="p">)</span> <span class="o">=&gt;</span> <span class="p">{</span>
  <span class="k">return</span> <span class="nx">JSON</span><span class="p">.</span><span class="nf">parse</span><span class="p">(</span><span class="k">new</span> <span class="nc">Buffer</span><span class="p">(</span><span class="nx">data</span><span class="p">,</span> <span class="dl">'</span><span class="s1">base64</span><span class="dl">'</span><span class="p">).</span><span class="nf">toString</span><span class="p">());</span>
<span class="p">}</span>

<span class="c1">// createSlackMessage create a message from a build object.</span>
<span class="kd">const</span> <span class="nx">createSlackMessage</span> <span class="o">=</span> <span class="p">(</span><span class="nx">build</span><span class="p">)</span> <span class="o">=&gt;</span> <span class="p">{</span>
  <span class="kd">let</span> <span class="nx">message</span> <span class="o">=</span> <span class="p">{</span>
   <span class="na">text</span><span class="p">:</span> <span class="s2">`Build </span><span class="se">\`</span><span class="p">${</span><span class="nx">build</span><span class="p">.</span><span class="nx">id</span><span class="p">}</span><span class="se">\`</span><span class="s2">`</span><span class="p">,</span>
    <span class="na">mrkdwn</span><span class="p">:</span> <span class="kc">true</span><span class="p">,</span>
    <span class="na">attachments</span><span class="p">:</span> <span class="p">[</span>
      <span class="p">{</span>
        <span class="na">title</span><span class="p">:</span> <span class="dl">'</span><span class="s1">Build logs - Your Custom Message Goes Here</span><span class="dl">'</span><span class="p">,</span>
        <span class="na">title_link</span><span class="p">:</span> <span class="nx">build</span><span class="p">.</span><span class="nx">logUrl</span><span class="p">,</span>
        <span class="na">fields</span><span class="p">:</span> <span class="p">[{</span>
          <span class="na">title</span><span class="p">:</span> <span class="dl">'</span><span class="s1">Status</span><span class="dl">'</span><span class="p">,</span>
          <span class="na">value</span><span class="p">:</span> <span class="nx">build</span><span class="p">.</span><span class="nx">status</span>
        <span class="p">}]</span>
      <span class="p">}</span>
    <span class="p">]</span>
  <span class="p">};</span>
  <span class="k">return</span> <span class="nx">message</span>
<span class="p">}</span></code></pre></figure>

<h2 id="3-deploy-the-cloud-function">3. Deploy the Cloud Function</h2>

<p>To deploy the subscribe function with a Cloud Pub/Sub trigger, run the following command in the <code class="language-plaintext highlighter-rouge">gcb_slack</code> directory:</p>

<figure class="highlight"><pre><code class="language-bash" data-lang="bash">gcloud functions deploy subscribe <span class="nt">--stage-bucket</span> <span class="o">[</span>STAGING_BUCKET_NAME] <span class="se">\</span>
    <span class="nt">--trigger-topic</span> cloud-builds</code></pre></figure>

<p>where <code class="language-plaintext highlighter-rouge">[STAGING_BUCKET_NAME]</code> is the name of your staging Cloud Storage Bucket that you defined earlier.</p>

<p>You should see an output confirming the creation of the cloud function and <code class="language-plaintext highlighter-rouge">status: READY</code>.</p>

<p>After you&#8217;ve completed deployment of the Cloud Function, when a build event occurs, you will receive a Slack notification.</p>

<p><img src="/assets/images/posts-images/2019-03-01-slack-app.png" alt="The Slack App in action" /></p>
<figcaption class="caption">The Slack App in action.</figcaption>
<p><br /></p>

<p>Also, feel free to customize your app, like adding a custom icon, as I did with mine. &#9757;&#65039;</p>

<hr />]]></content><author><name>Rustam Mehmandarov</name></author><summary type="html">Setting up Slack notifications for Google Cloud Build jobs with code examples and screenshots.</summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://mehmandarov.com/assets/images/posts-images/lego-bots.jpg"/><category term="blog"/><category term="field notes"/><category term="automation"/><category term="cloud"/><category term="english"/></entry><entry><title type="html">Computers as Movie Superstars</title><link href="https://mehmandarov.com/computer-superstars/" rel="alternate" type="text/html" title="Computers as Movie Superstars"/><published>2019-02-19T09:01:00+01:00</published><updated>2019-02-19T09:01:00+01:00</updated><id>https://mehmandarov.com/computer-superstars</id><content type="html" xml:base="https://mehmandarov.com/computer-superstars/"><![CDATA[<p><em>The untold story of a short movie for a big conference. You might have seen all the Easter eggs, but I am sure you didn&#8217;t know this! These are some the lesser-known facts about some of the computers used in Stranger Strings made for JavaZone 2018.</em></p>

<ul>
  <li><a href="#the-beginning">The Beginning</a></li>
  <li><a href="#the-new-level">The New Level</a></li>
  <li><a href="#story-1-an-icon-of-norwegian-it-history">Story 1: An Icon of Norwegian IT History</a></li>
  <li><a href="#story-2-a-machine-for-4300-usd">Story 2: A Machine for 4.300 USD</a></li>
  <li><a href="#story-3-two-portable-computers">Story 3: Two &#8220;Portable&#8221; Computers</a></li>
  <li><a href="#story-4-a-very-special-machine">Story 4: A Very Special Machine</a></li>
  <li><a href="#stranger-strings-the-movie">Stranger Strings: The Movie</a></li>
</ul>

<hr />

<p>You may already have seen <em>Stranger Strings</em>, a tribute to the <em><a href="https://www.imdb.com/title/tt4574334/">Stranger Things</a></em> series, that was created for JavaZone 2018. This was our latest addition to the long list of short films we have created throughout the years.</p>

<p>Here, I would like to share the untold story of the computers and hardware from the movie that might have been seen merely as the props in the movie to the untrained eye, but when you look a bit deeper have quite interesting stories to tell. <a href="https://javazone.no">JavaZone</a> has a tradition of creating short videos before the conference and most of the videos &#8211; with a very few exceptions &#8211; are in the format of trailers or short movies. These are usually geeky parodies to well-known (and usually well-loved) movies or series. This time we decided to do something a bit different. We created a slightly longer film showing us a parallel, and geekier, <em>Stranger Things</em> universe, and called it <em>Stranger Strings</em>.</p>

<p><img src="/assets/images/posts-images/2019-02-03-stranger-strings1.png" alt="Stranger Strings" /></p>
<figcaption class="caption">Stranger Strings.</figcaption>
<p><br /></p>

<h2 id="the-beginning">The Beginning</h2>
<h3 id="easter-eggs-gotta-collect-em-all">Easter Eggs: Gotta Collect &#8216;em All!</h3>

<p>We had an amazing JavaZone team, film crew, and actors. It was great fun being a part of the team creating the video and we worked hard on trying to add as many fun references and puns as possible without going over the top. We also did include a hint to a mini online adventure game made by our team and a puzzle that would lead you to a <em>surprise</em> (I still won&#8217;t disclose that one!). In case you haven&#8217;t seen it yet, you will find a link to the game and hints by just watching the movie a little bit closer.</p>

<p><img src="/assets/images/posts-images/2019-02-03-stranger-strings2.jpg" alt="Picture from Stranger Strings movie set. Photo: Rustam Mehmandarov" /></p>
<figcaption class="caption">Picture from Stranger Strings movie set. Photo: Rustam Mehmandarov</figcaption>
<p><br /></p>

<p>Some of these things were right out in the open, in plain sight, while others were much subtler. Nonetheless, with 1.000 views a day on the first week, and a bit short of 20.000 views in total as of today, most of the Easter eggs and references were noticed by quite a few people.</p>

<h2 id="the-new-level">The New Level</h2>

<p>However, the movie still has something else that was a bit more difficult to notice simply by watching it, and I think it is finally about time to reveal those things as well.</p>

<p>When setting up the set for this movie one of the things we had to do was to find computers from the correct time period, and we needed quite a few of them. So, after asking around on social media, we were able to get hold of a few machines. And what machines they were!</p>

<p><img src="/assets/images/posts-images/2019-02-03-tweet.png" alt="The Tweet &#8211; looking for old computers in Oslo" /></p>
<figcaption class="caption">The Tweet &#8211; looking for old computers in Oslo.</figcaption>
<p><br /></p>

<p>This is where it all started, and where our movie for 2018 got yet another layer of hidden features.</p>

<p>We got all the machines from the collection of the University of Oslo Library, Department of Informatics (<em>again, thanks a lot for providing those!</em>). And all these machines had quite an interesting story to tell.</p>

<p>So, in addition to serving the main purpose of being, well, the machines from the time period for the movie set, they also brought a tiny bit of the Norwegian computer history to the movie.</p>

<p>(<em><strong>Spoiler alert</strong>: One could argue that at least one of the machines had a role in the development of <a href="https://en.wikipedia.org/wiki/Simula">Simula</a>. Excited to know which, or what even Simula is? Read on!</em>)</p>

<p>Now, let&#8217;s have a look at some of the highlights of what we had at the set.</p>

<h2 id="story-1-an-icon-of-norwegian-it-history">Story 1: An Icon of Norwegian IT History</h2>
<h3 id="control-panel-for-nord-10s-norsk-data">Control Panel for NORD-10/S (Norsk Data)</h3>
<p>The first one out is the control panel from Norsk Data&#8217;s <a href="https://en.wikipedia.org/wiki/Nord-10">NORD-10/S</a>. We used it as a part of the &#8220;mainframe&#8221; in the movie. It might not have been very visible in the movie, but it still had a great symbolic value.</p>

<p><img src="/assets/images/posts-images/2019-02-03-mainframe1.png" alt="The mainframe (screenshot from the movie)" /></p>
<figcaption class="caption">The mainframe (screenshot from the movie).</figcaption>
<p><br /></p>

<p>It was great to be able to include a piece of that history into the Stranger Strings. What makes this piece special is that the company had quite a significant role in the Norwegian IT history.</p>

<p>There are quite a few interesting facts connected to <a href="https://no.wikipedia.org/wiki/Norsk_Data">Norsk Data</a>. It was a significant supplier of minicomputers to many research projects, in particular to CERN in Geneva, Switzerland. It was also the first Norwegian company to get listed on the London Stock Exchange (1981) and NASDAQ (1983) &#8211; just to mention a few.</p>

<p><img src="/assets/images/posts-images/2019-02-03-mainframe2.jpg" alt="The mainframe (behind the scenes). Photo: Rustam Mehmandarov" /></p>
<figcaption class="caption">The mainframe (behind the scenes). Photo: Rustam Mehmandarov</figcaption>
<p><br /></p>

<p>Also, this particular piece of hardware was acquired by the University of Oslo in 1979 for the university&#8217;s main library located off the new, main campus, <a href="https://www.ub.uio.no/om/tall-og-fakta/ub-200-aar/bibliotek/solli.html">at Drammensveien</a> (Norwegian article). It came with all the cool stuff that might want back then, like the state-of-the-art punch card reader, and it was used to connect the library with a direct line to the rest of the university network.</p>

<h2 id="story-2-a-machine-for-4300-usd">Story 2: A Machine for 4.300 USD</h2>
<h3 id="atari-mega-st2">Atari Mega ST2</h3>
<p>Now, let&#8217;s talk about Will&#8217;s machine &#8211; an Atari Mega ST2. It was packing an 8 MHz Motorola 68000 processor, had a whopping 2 MB RAM, and 20 MB hard drive.</p>

<p>This machine was one of the 35 machines bought by the Department of Informatics at the University of Oslo in May 1988 and came with an impressive price tag of 18.360 NOK a piece (which corresponds to 36.269 NOK, or 4.297 USD in today&#8217;s equivalent).</p>

<p><img src="/assets/images/posts-images/2019-02-03-atari.png" alt="Atari Mega ST2 (screenshot from the movie)" /></p>
<figcaption class="caption">Atari Mega ST2 (screenshot from the movie).</figcaption>
<p><br /></p>

<p>So, we can surely say that this machine did play a role in the education of a few generations of IT professionals here in Norway, before starring in a movie exactly 30 years later.</p>

<h2 id="story-3-two-portable-computers">Story 3: Two &#8220;Portable&#8221; Computers</h2>
<h3 id="ibm-portable-personal-computer">IBM Portable Personal Computer</h3>

<p>We also got to borrow not one, but <em>two</em> of IBM&#8217;s portable personal computers. This model, form 1984, was the IBM&#8217;s first portable machine, had an 8088 processor, and weighed approximately 15 kg (33 lbs).</p>

<p><img src="/assets/images/posts-images/2019-02-03-ibm-portable1.jpg" alt="Two IBM Portable Personal Computers, with IBM XT in the middle (picture from the movie set). Photo: Rustam Mehmandarov" /></p>
<figcaption class="caption">Two IBM Portable Personal Computers, with IBM XT in the middle (picture from the movie set). Photo: Rustam Mehmandarov.</figcaption>
<p><br /></p>

<p>Yes, portable meant something else back in the days&#8230;</p>

<p><img src="/assets/images/posts-images/2019-02-03-ibm-portable2.png" alt="A very light and portable IBM Portable Personal Computer (screenshot from the movie)" /></p>
<figcaption class="caption">A very light and portable IBM Portable Personal Computer (screenshot from the movie).</figcaption>
<p><br /></p>

<h2 id="story-4-a-very-special-machine">Story 4: A Very Special Machine</h2>
<h3 id="ibm-xt">IBM XT</h3>
<p>I saved the best for last! The machine you could often see in the movie placed between two portable IBMs, and the main machine of one of the main characters &#8211; Three &#8211; is an <a href="https://en.wikipedia.org/wiki/IBM_Personal_Computer_XT">IBM XT</a>.</p>

<p>IBM XT was one of IBM&#8217;s first PC models. It came with two floppy drives for 5.25&#8221; disks and 256 KB RAM. This particular machine was later upgraded to 640 KB RAM and one of the floppy drives was replaced with a 20 MB hard drive.</p>

<p><img src="/assets/images/posts-images/2019-02-03-ibm-xt1.jpg" alt="IBM XT, with IBM Portable Personal Computer in the background (picture from the movie set). Photo: Rustam Mehmandarov" /></p>
<figcaption class="caption">IBM XT, with IBM Portable Personal Computer in the background (picture from the movie set). Photo: Rustam Mehmandarov.</figcaption>
<p><br /></p>

<p>So, what is so important about this machine, you might ask?</p>

<p>The most interesting thing about this particular machine is that it belonged to <a href="https://en.wikipedia.org/wiki/Kristen_Nygaard">Kristen Nygaard</a> &#8211; the co-inventor of object-oriented programming and the programming language Simula, together with <a href="https://en.wikipedia.org/wiki/Ole-Johan_Dahl">Ole-Johan Dahl</a>.</p>

<p><a href="https://en.wikipedia.org/wiki/Simula">Simula</a> was the first object-oriented programming language and the language that is considered to have had a strong influence on Java and many other modern object-oriented languages.</p>

<p>So, one can argue that this machine, being the office computer of Kristen Nygaard, had a direct role in the development of Simula and work related to it.</p>

<p><img src="/assets/images/posts-images/2019-02-03-ibm-xt2.jpg" alt="IBM XT (picture from the movie set). Photo: Rustam Mehmandarov" /></p>
<figcaption class="caption">IBM XT (picture from the movie set). Photo: Rustam Mehmandarov.</figcaption>
<p><br /></p>

<p><em>Finally, I would like to yet again thank everyone involved in the creation of this movie! You are all awesome and this would not have been as fun and great without you!</em></p>

<h1 id="stranger-strings-the-movie">Stranger Strings: The Movie</h1>

<p>Now, that you have read the stories behind some of the machines in the movie, it is time to watch it again.</p>

<iframe width="560" height="315" src="https://www.youtube.com/embed/EAH3i6l8PbY" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen=""></iframe>

<hr />]]></content><author><name>Rustam Mehmandarov</name></author><summary type="html">The untold story behind the vintage computers used in the JavaZone 2018 short film Stranger Strings.</summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://mehmandarov.com/assets/images/posts-images/old-camera.jpg"/><category term="blog"/><category term="javazone"/><category term="english"/></entry><entry><title type="html">Disposable Docker Containers</title><link href="https://mehmandarov.com/disposable-docker-containers/" rel="alternate" type="text/html" title="Disposable Docker Containers"/><published>2019-01-01T11:01:00+01:00</published><updated>2019-01-01T11:01:00+01:00</updated><id>https://mehmandarov.com/disposable-docker-containers</id><content type="html" xml:base="https://mehmandarov.com/disposable-docker-containers/"><![CDATA[<p><em>Disposable containers may sound like a tautology. However, here we will be looking into single-use, ephemeral containers &#8211; even in the context of the containers &#8211; that are used for building and testing applications, and disposing of them shortly after.</em></p>

<hr />

<p>Containers are something that we use to run our applications and, normally, we dispose of the whole container when we build a new version of the application or need to upgrade something in the setup. This means that containers are generally having a short lifespan.</p>

<p>However, in this case, I want to show you how to build something that exists for an even shorter period of time and that can be used as an alternative to a local setup for building and testing applications locally before pushing it to test, staging, production, etc.</p>

<p>This is a simplified example of what is being done on a much bigger scale with moving your CI/CD pipelines to such disposable containers, and with libraries like <a href="https://www.testcontainers.org/">Testcontainers</a>.</p>

<p>In this case, I would like to show you how to setup Jekyll applications, but this can be easily applied to any kind of applications written in any of your favorite languages, like Java or Python. Until recently, I have been running a Jekyll installation locally with all dependencies installed on my machine. However, it has been a bit challenging when moving between machines and reinstalling operating systems. To simplify the process, I decided to containerize the local build and test processes.</p>

<p>I wanted the following:</p>

<ul>
  <li>To build my code from and to the local folder on my (host) machine</li>
  <li>Run the application (in this case this blog) from a local folder on my (host) machine</li>
  <li>Avoid setting up the environment, or have a minimal and portable setup</li>
  <li>Avoid environment clean-up &#8211; I didn&#8217;t want to hold on to the unnecessary containers and container images</li>
</ul>

<h2 id="tldr-the-solution"><em><strong>TL;DR</strong></em>: The solution</h2>
<p><em><strong>(see next section for the explanation)</strong></em></p>

<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span><span class="nb">export </span><span class="nv">JEKYLL_VERSION</span><span class="o">=</span>3.8
<span class="nv">$ </span>docker run <span class="nt">--rm</span> <span class="nt">--volume</span><span class="o">=</span><span class="s2">"</span><span class="nv">$PWD</span><span class="s2">:/srv/jekyll"</span> <span class="se">\</span>
       <span class="nt">-it</span> jekyll/jekyll:<span class="nv">$JEKYLL_VERSION</span> jekyll build
<span class="nv">$ </span>docker run <span class="nt">--name</span> newblog <span class="nt">--volume</span><span class="o">=</span><span class="s2">"</span><span class="nv">$PWD</span><span class="s2">:/srv/jekyll"</span> <span class="nt">-p</span> 4000:4000 <span class="se">\</span>
       <span class="nt">-it</span> jekyll/jekyll:<span class="nv">$JEKYLL_VERSION</span> jekyll serve <span class="nt">--watch</span> <span class="nt">--drafts</span></code></pre></figure>

<h2 id="explanation--line-by-line">Explanation &#8211; line by line</h2>

<p>So, let&#8217;s take a closer look at each of the lines:</p>

<figure class="highlight"><pre><code class="language-bash" data-lang="bash">1: <span class="nb">export </span><span class="nv">JEKYLL_VERSION</span><span class="o">=</span>3.8</code></pre></figure>

<p>Just setting up versions that will be used later &#8211; a bit of housekeeping. Nothing exciting here.</p>

<hr />

<figure class="highlight"><pre><code class="language-bash" data-lang="bash">2: docker run <span class="nt">--rm</span> <span class="nt">--volume</span><span class="o">=</span><span class="s2">"</span><span class="nv">$PWD</span><span class="s2">:/srv/jekyll"</span> <span class="se">\</span>
        <span class="nt">-it</span> jekyll/jekyll:<span class="nv">$JEKYLL_VERSION</span> jekyll build</code></pre></figure>

<p>Here, we build the code and output it to the same disk volume as the source code, i.e. the volume that is shared with my host machine. Now I have the built version on my machine without the hassle of setting up the local build environment. In addition to that, I will be doing some clean-up, by deleting the build container after the build job is finished.</p>

<ul>
  <li><code class="language-plaintext highlighter-rouge">--rm</code> &#8211; just execute the command and clean-up (remove the container, file system, etc.)</li>
  <li><code class="language-plaintext highlighter-rouge">--volume</code> &#8211; mapping the current directory to <code class="language-plaintext highlighter-rouge">/srv/jekyll</code> in the container</li>
  <li><code class="language-plaintext highlighter-rouge">-it</code> instructs Docker to allocate a pseudo-TTY connected to the container&#8217;s stdin; creating an interactive shell in the container
    <ul>
      <li><code class="language-plaintext highlighter-rouge">-i</code> &#8211; attach container&#8217;s STDIN</li>
      <li><code class="language-plaintext highlighter-rouge">-t</code> &#8211; allocate a pseudo-TTY</li>
    </ul>
  </li>
  <li><code class="language-plaintext highlighter-rouge">jekyll/jekyll:$JEKYLL_VERSION</code> &#8211; Docker <a href="https://github.com/envygeeks/jekyll-docker/blob/master/README.md">image</a> to use and the tag</li>
  <li><code class="language-plaintext highlighter-rouge">jekyll build</code> &#8211; command to run</li>
</ul>

<hr />

<figure class="highlight"><pre><code class="language-bash" data-lang="bash">3. docker run <span class="nt">--name</span> newblog <span class="nt">--volume</span><span class="o">=</span><span class="s2">"</span><span class="nv">$PWD</span><span class="s2">:/srv/jekyll"</span> <span class="nt">-p</span> 4000:4000 <span class="se">\</span>
        <span class="nt">-it</span> jekyll/jekyll:<span class="nv">$JEKYLL_VERSION</span> jekyll serve <span class="nt">--watch</span> <span class="nt">--drafts</span></code></pre></figure>

<p>This will create another container that will be running our application. Here we will need to add a few other parameters &#8211; like mapping the container ports to the ports on the local machine and giving the container a name.</p>

<ul>
  <li><code class="language-plaintext highlighter-rouge">--name newblog</code> &#8211; give your container a name</li>
  <li><code class="language-plaintext highlighter-rouge">--volume</code> &#8211; mapping the current directory to <code class="language-plaintext highlighter-rouge">/srv/jekyll</code> in the container</li>
  <li><code class="language-plaintext highlighter-rouge">-p</code> &#8211; bind port 4000 of the container to TCP port 4000 (<code class="language-plaintext highlighter-rouge">-p host_machine:container</code>)</li>
  <li><code class="language-plaintext highlighter-rouge">-it</code> instructs Docker to allocate a pseudo-TTY connected to the container&#8217;s stdin; creating an interactive shell in the container
    <ul>
      <li><code class="language-plaintext highlighter-rouge">-i</code> &#8211; attach container&#8217;s STDIN</li>
      <li><code class="language-plaintext highlighter-rouge">-t</code> &#8211; allocate a pseudo-TTY</li>
    </ul>
  </li>
  <li><code class="language-plaintext highlighter-rouge">jekyll/jekyll:$JEKYLL_VERSION</code> &#8211; Docker <a href="https://github.com/envygeeks/jekyll-docker/blob/master/README.md">image</a> to use and the tag</li>
  <li><code class="language-plaintext highlighter-rouge">jekyll serve --watch --drafts</code> &#8211; command to run</li>
</ul>

<p>Now you can stop the container with <code class="language-plaintext highlighter-rouge">CTRL+c</code>, and restart it again with:</p>

<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span>docker start newblog <span class="nt">-i</span></code></pre></figure>

<p>If you don&#8217;t want the container being persistent on your system, you can simply add <code class="language-plaintext highlighter-rouge">--rm</code> as in the previous command:</p>

<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span>docker run <span class="nt">--rm</span> <span class="nt">--name</span> newblog <span class="nt">--volume</span><span class="o">=</span><span class="s2">"</span><span class="nv">$PWD</span><span class="s2">:/srv/jekyll"</span> <span class="nt">-p</span> 4000:4000 <span class="se">\</span>
       <span class="nt">-it</span> jekyll/jekyll:<span class="nv">$JEKYLL_VERSION</span> jekyll serve <span class="nt">--watch</span> <span class="nt">--drafts</span></code></pre></figure>

<hr />]]></content><author><name>Rustam Mehmandarov</name></author><summary type="html">Using single-use, ephemeral Docker containers for building and testing applications locally.</summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://mehmandarov.com/assets/images/posts-images/containers.jpg"/><category term="blog"/><category term="containers"/><category term="docker"/><category term="jekyll"/><category term="field notes"/><category term="english"/></entry><entry><title type="html">Navigating and Editing the Command Line &#8211;&#160;Bash Edition</title><link href="https://mehmandarov.com/navigating-and-editing-the-command-line/" rel="alternate" type="text/html" title="Navigating and Editing the Command Line &#8211;&#160;Bash Edition"/><published>2018-12-30T21:01:00+01:00</published><updated>2018-12-30T21:01:00+01:00</updated><id>https://mehmandarov.com/navigating-and-editing-the-command-line</id><content type="html" xml:base="https://mehmandarov.com/navigating-and-editing-the-command-line/"><![CDATA[<p><em>A cheat sheet for moving around and editing your command line &#8211;&#160;Bash Edition.</em></p>

<ul>
  <li><a href="#moving-around-the-command-line">Moving Around the Command Line</a></li>
  <li><a href="#editing-commands-in-the-command-line">Editing Commands in the Command Line</a></li>
  <li><a href="#bonus">Bonus</a></li>
</ul>

<hr />

<p>Using the command line can simplify and even automate many of the operations we do on a computer. However, using the command line can mean quite a bit of typing and a possibly large number of parameters. In this post, I would like to focus on how to navigate the cursor and edit the command line, while leaving all the other Bash tricks for the future posts.</p>

<p>I also have created simple graphics to illustrate some of the main shortcuts listed below. This (hi-res) image can be printed for future reference.</p>

<p><em>Note:</em> Please note that all commands containing <code class="language-plaintext highlighter-rouge">ALT</code> combinations might not work depending on your system configuration, and most definitely not work on MacOS. Normally, it is because these combinations are mapped to something else. However, you can still use the same shortcuts simply by replacing <code class="language-plaintext highlighter-rouge">ALT</code> with <code class="language-plaintext highlighter-rouge">ESC</code>.</p>

<p><img src="/assets/images/posts-images/2018-12-30-bash_navigation.png" alt="Navigating and Editing the Command Line (Bash Edition)" /></p>
<figcaption class="caption"> Navigating and Editing the Command Line (Bash Edition)</figcaption>

<hr />

<h2 id="moving-around-the-command-line">Moving Around the Command Line</h2>

<p>So, let&#8217;s first speak about how to move the cursor around &#8211; because using just arrow keys is often not the most optimal way of navigating. Sometimes you might want to go to the beginning of the line, to the end of the line, or simply jump from one <em>word</em> to another, where <em>word</em> &#8211; in this context &#8211; is set of characters separated by spaces (or sometimes other special characters), or as <a href="https://linux.die.net/man/1/bash">documentation</a> states it:</p>

<blockquote>
  <p>A sequence of characters considered as a single unit by the shell. Also known as a token.</p>
</blockquote>

<figure class="highlight"><pre><code class="language-text" data-lang="text"># Moving the cursor &#8211; fast
CTRL+a         Go to the beginning of the line (same as Home)
CTRL+e         Go to the End of the line (same as End)
ALT+b / ESC+b  Go one word back (to the left)
ALT+f / ESC+f  Go one word forward (to the right)

# Moving the cursor &#8211;&#160;one character at a time
CTRL+f         Go forward one character
CTRL+b         Go backward one character

# Using history
CTRL+r         Backwards search in previously executed commands (history)
CTRL+p         Previous command (same as Up arrow)
CTRL+n         Next command (same as Down arrow)</code></pre></figure>

<hr />

<h2 id="editing-commands-in-the-command-line">Editing Commands in the Command Line</h2>

<p>Now that we are able to navigate freely along the command line, it is time to do some modifications. Here, we will see how to delete, cut, paste, and swap words and characters.</p>

<figure class="highlight"><pre><code class="language-text" data-lang="text"># Deleting whole words
ALT+Del        Delete the word before (to the left of) the cursor
ALT+d / ESC+d  Delete the word after (to the right of) the cursor
CTRL+w         Cut the word before the cursor to the clipboard

# Deleting parts of the line
CTRL+k         Cut the line after the cursor to the clipboard
CTRL+u         Cut/delete the line before the cursor to the clipboard

# Deleting single characters
CTRL+d         Delete character under the cursor (same as Delete key)
CTRL+h         Delete character before the cursor (same as Backspace key)

# Paste, Undo, revert, and more
CTRL+l         Clear the screen (similar to the 'clear' command)
CTRL+y         Paste the last thing to be cut (yank)
CTRL+_         Undo
ALT+r / ESC+r  Revert the changes and replace with the line as it was 
                in History.

# Swap 'em!
CTRL+t         Swap the last two characters before the cursor
ALT+t / ESC+t  Swap current word with previous
 
# Convert to UPPER, lower, or Sentence case
ALT+u / ESC+u  Capitalise characters from the cursor to the end of 
                the current word and move to the end of the word.
ALT+l / ESC+l  Lower the case of characters from the cursor to the
                end of the current word and move to the end of the word.
ALT+c / ESC+c  Capitalize the character under the cursor position 
                and move to the end of the word.</code></pre></figure>

<hr />

<h2 id="bonus">Bonus</h2>

<p>First, the most obvious &#8211; you can always find more gems in the <code class="language-plaintext highlighter-rouge">man</code> pages for Bash both in your terminal and online (for instance on this <a href="https://linux.die.net/man/1/bash">mirror</a>). To view it in your terminal, type:</p>

<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span>man bash</code></pre></figure>

<p>Now, over to something different. Since we have been talking about the command line and shells it is worth mentioning some less-known (and sometimes <em>&#8220;as a curiosity&#8221;</em>) shortcuts in another terminal &#8211; Command Prompt, <code class="language-plaintext highlighter-rouge">cmd.exe</code>:</p>

<figure class="highlight"><pre><code class="language-text" data-lang="text">Function keys in cmd.exe:
  - F1: Pastes the last executed command (character by character)
  - F2: Pastes the last executed command (up to the entered character)
  - F3: Pastes the last executed command
  - F4: Deletes current prompt text up to the entered character
  - F5: Pastes recently executed commands (does not cycle)
  - F6: Pastes ^Z to the prompt
  - F7: Displays a selectable list of previously executed commands
  - F8: Pastes recently executed commands (cycles)
  - F9: Asks for the number of the command from the F7 list to paste</code></pre></figure>

<p><em>Good luck! Try them out and let me know how that goes!</em></p>

<hr />]]></content><author><name>Rustam Mehmandarov</name></author><summary type="html">A cheat sheet for navigating and editing the command line in Bash.</summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://mehmandarov.com/assets/images/posts-images/laptop.jpg"/><category term="blog"/><category term="command line"/><category term="bash"/><category term="field notes"/><category term="english"/></entry><entry><title type="html">Escaping Developer Nightmares</title><link href="https://mehmandarov.com/escaping-developer-nigthmares/" rel="alternate" type="text/html" title="Escaping Developer Nightmares"/><published>2017-12-01T07:23:00+01:00</published><updated>2017-12-01T07:23:00+01:00</updated><id>https://mehmandarov.com/escaping-developer-nigthmares</id><content type="html" xml:base="https://mehmandarov.com/escaping-developer-nigthmares/"><![CDATA[<p><em>A short write up of the bad things we do in software development and some suggestions on how to fix them.</em></p>

<ul>
  <li><a href="#the-existing-state-of-affairs">The Existing State of Affairs</a></li>
  <li><a href="#the-moving-parts">The Moving Parts</a></li>
  <li><a href="#conclusion">Conclusion</a></li>
</ul>

<hr />

<p>Let&#8217;s take a look into what we can do to achieve a better development environment than an average development project &#8211; a project that most of us have seen at some point in our professional lives, or maybe even are a part of right now. We will also look into some tools and patterns that will help us convert those projects into a paradise for the developers.</p>

<p>Just a few decades ago, we were working in ways that might look like unproductive, in the best case. Our development models were predominated by waterfalls, our IDEs were basic and we were compiling our projects by hand, using <code class="language-plaintext highlighter-rouge">javac</code>, or building up the <code class="language-plaintext highlighter-rouge">CLASSPATH</code> depending on the <code class="language-plaintext highlighter-rouge">GOTO</code> statements in a huge spaghetti code contained in countless <code class="language-plaintext highlighter-rouge">bat</code> files. Our code lived in a very simple versioning systems that were not distributed or supported branching strategies that are praised by the developers today. Our documentation lived in <code class="language-plaintext highlighter-rouge">doc</code> files on shared network drives, side by side with the simple issue tracking systems, that don&#8217;t even get close to what we have today.</p>

<p>Today, it is all different &#8211; we have Git, real issue tracking, IDEs, all that integrated with build servers and collaborative platforms. Yes, everything is much better, more effective and user-friendly, one might think that we are in the paradise already? Well&#8230; yes, things are fortunately getting better, however, we are still doing things in a way that might still give you nightmares, several decades from now.</p>

<p>Last years I have been working and invited to evaluate and help with an audit of various projects. Here are some of my observations and thoughts.</p>

<h2 id="the-existing-state-of-affairs">The Existing State of Affairs</h2>

<p>This <a href="https://twitter.com/brianwisti/status/503987766032494592?lang=en">tweet</a> describes it pretty well:</p>

<figure class="highlight"><pre><code class="language-text" data-lang="text">YOU ARE IN A LEGACY CODEBASE
&gt; RUN TESTS
YOU HAVE NO TESTS
&gt; READ SPEC
YOU HAVE NO SPEC
&gt; WRITE FIX
YOU ARE EATEN BY AN ELDER CODE HACK.</code></pre></figure>

<p>Some of the issues are, naturally, remnants of the past &#8211; the legacy systems; but even those systems and most of the other problems we see today can be avoided if we slightly change our view at some of the main parts of the development process. In most of the cases, we would be aware of those issues, but we might need to explain and motivate the others &#8211; often people are responsible for the projects and those who prioritize the development and maintenance backlog.</p>

<h2 id="the-moving-parts">The Moving Parts</h2>

<p>The road to a great nightmare-free future consists of three components: the code quality, the development and build tools, and a good documentation and collaboration systems. When evaluating systems I often start asking some simple questions listed below to get an idea of the system.</p>

<h3 id="1-the-code">1. The Code</h3>
<h4 id="the-code-quality">The Code Quality</h4>

<p>First thing off is the general code quality. I often start by asking about simple things &#8211; if the project has a coding standard, and if it is being followed. I also ask to take a quick peek at the code and check minor things like file encodings and MIME types. I also follow up with a question if the team is practicing code reviews, and how they are doing that.</p>

<p>While those things alone don&#8217;t have to mean anything, and are minor issues individually, together with other factors they still are initial indicators of possible neglect. This gives me a possibility to map areas where to look further.</p>

<p>In addition to that, there are also some more specific parts that will be listed as sub-sections below.</p>

<h4 id="the-third-party-libraries">The third-party libraries</h4>
<p>The role of the third party libraries and their use is often forgotten and neglected when considering code and system quality. This is quite unfortunate as this is the part of the code that you might not be able to patch easily, and is harder to maintain compared to your own codebase. Here are some simple questions that might help with getting a better grip on third-party libraries:</p>

<ul>
  <li>Do you keep track of your third-party libraries?</li>
  <li>Do you regularly check if there are known issues or vulnerabilities in them?</li>
  <li>Do you have a plan for keeping them updated?</li>
  <li>Are the libraries you are using being actively maintained by the authors?</li>
  <li>Are the libraries you are using compatible with each other?</li>
  <li>Do the libraries you are using have appropriate licenses that are compatible with your system? <em>(This also applies to the open source software licenses.)</em></li>
</ul>

<p>Issues and vulnerabilities are being found and patched all the time. As an example for this, let me point to Google&#8217;s <a href="https://github.com/google/oss-fuzz">OSS-Fuzz Project</a> that has found <a href="https://testing.googleblog.com/2017/05/oss-fuzz-five-months-later-and.html">numerous security vulnerabilities</a> in several critical open source projects. Unfortunately, even though many people are aware of the security issues in software in general, the library updates still often tend to be forgotten.</p>

<p>It is also worth noting that while most of the issues on my list above are security related, the last one might be of a legal sort, and probably is the most neglected of the issues listed.</p>

<h4 id="the-architecture">The Architecture</h4>

<p>I am often being asked to assess a system and tell something about its architecture compared to more modern systems. The different aspects of the system&#8217;s architecture will tell a lot about its maintainability both when it comes to further development, bug fixing, and keeping the system running. Some questions that might help with determining the state of the system would be:</p>

<ul>
  <li>Does your architecture support automated deployment?</li>
  <li>Does your architecture support continuous deploy and delivery?</li>
  <li>Does your architecture support load balancing?</li>
  <li>Does your architecture support microservices?</li>
  <li>How is the architecture implemented in the code?</li>
</ul>

<h4 id="tools-for-maintaining-the-code-quality">Tools for Maintaining the Code Quality</h4>

<p>Some of this might sound familiar to you when you think of one or several systems you had been working with and you might now be wondering what you can do to improve the code quality? Further steps here would be starting to use proper tools that will be able to tell more about the various aspects of your code. Some of them can be used as plugins to your build system (like Maven), and some could be stand-alone tools.</p>

<p>Stand-alone tools for code analysis:</p>
<ul>
  <li><a href="https://www.sonarqube.org/">SonarQube</a></li>
  <li><a href="https://pmd.github.io/">PMD</a></li>
  <li><a href="http://findbugs.sourceforge.net/">FindBugs</a></li>
</ul>

<p>Maven plugins to consider (more about plugins in my <a href="/five-pillars-of-a-good-maven-project/">previous post</a>):</p>
<ul>
  <li>Assembly</li>
  <li>Versions</li>
  <li>Dependency</li>
  <li>Enforcer</li>
  <li>Surefire</li>
  <li>Failsafe</li>
  <li>Sonar</li>
  <li>Findbugs</li>
  <li>pmd</li>
</ul>

<p><em><strong>Bonus:</strong></em> See this post on <a href="/cmd-tools-for-developers/">command line tools for Java projects</a>.</p>

<hr />

<h3 id="2-development-tools-and-strategies">2. Development Tools and Strategies</h3>

<p>Now, let&#8217;s talk about the development tools. All that fancy code and great architecture will not bring you any closer to a developer&#8217;s paradise if there will not be some proper tools to support the development. The code should live in a proper version control system that supports collaboration and things like branching and tagging. There should also be tools that help you with code quality analysis, static code analysis, etc. A good starting point here would be to start with answering the following about the project in question:</p>

<ul>
  <li>Do you use a proper code versioning tool &#8211; Git, or even SVN?</li>
  <li>Do you have a branching (and tagging) strategy?</li>
  <li>Do you have a way of measuring code complexity?</li>
  <li>Do you have a way of measuring test coverage and results?</li>
  <li>Do you run static code analysis?</li>
</ul>

<p>Some tools that can help you here (again, for Java-based systems):</p>

<ul>
  <li><strong>IDEs and IDE plug-ins</strong> that can do checks at commits, integrate with test and QA tools, etc.</li>
  <li><strong>Build tools:</strong> Maven, Gradle, etc.</li>
  <li><strong>Continuous integration tools:</strong> Jenkins, TeamCity, Bamboo, etc.</li>
  <li><strong>Frameworks and tools for testing</strong>: to run unit tests, integration tests, UI tests, and end-to-end tests.</li>
</ul>

<p>There are some further strategies and questions to consider:</p>

<ul>
  <li>Are your environments easy to reproduce with minimal efforts &#8211; can you rebuild it by simply running a script?</li>
  <li>Do you have a proper pipeline from <em>packaging</em>, to <em>delivery</em>, to <em>deploy</em>?</li>
  <li>To what environments can you deploy automatically? With the same script, or command?</li>
  <li>Are your environments (like <em>development</em>, <em>testing</em>, <em>staging</em>, <em>pre-prod</em>, <em>production</em>) similar to each other?</li>
  <li>Do you follow the same process to deploy to each environment?</li>
  <li>Are <em>QA</em> and <em>production</em> running on physically separate hardware?</li>
  <li>Are you monitoring all of the environments? (i.e. are you able to see errors before they make it to <em>QA</em> or even <em>production</em>?)</li>
</ul>

<hr />

<h3 id="3-documentation-and-collaboration-tools">3. Documentation and Collaboration Tools</h3>

<p>Last, but not least, we will need to talk about the tools for collaboration and documentation. Without these tools, we will be back to the way things were several decades ago &#8211; with documents on shared network drives and other horrors of the 90&#8217;s that I mentioned at the beginning of this post. However, good wikis, other collaboration tools, and proper issue tracking will bring your software to another level, encouraging continuous improvement of the system.</p>

<ul>
  <li>Wiki</li>
  <li>Collaboration &#8211; chat, etc.</li>
  <li>Issue tracking tools</li>
</ul>

<p>No matter how obvious it might seem, it is still important to note that one should avoid multiple documentation and issue tracking systems. Unfortunately, even though it might sound obvious, it is more common than you think &#8211; I have seen my share of systems for documentation and issue tracking resulting in fragmented information and confusion.</p>

<h2 id="conclusion">Conclusion</h2>

<p>There are several challenges connected with having and maintaining the good code quality. The first challenge is that a good code quality is not something you can achieve overnight. It takes time and energy to achieve that and it is a continuous process. You will need some tools, techniques, and methodology to prevail, and it will probably be easier to introduce all that from the beginning of a project.</p>

<p>The second challenge would be that it might be hard to convince the stakeholders of the project to invest time and resources into something that does not bring any visible improvements to the table &#8211; things like new features and bug fixes are more likely to get prioritized over something that cannot be easily measured.</p>

<p>Actually, while presenting on this topic at JavaOne 2017 in San Francisco, several of the attendees asked me about the ways of getting to a beautiful nightmare-free code and infrastructure, and the ways of convincing the stakeholders that this is the way to go. Unfortunately, there is no one simple solution to this, and the most valuable thing, in this case, would be to show the real value of the good quality code.</p>

<p>The measurements parameters to show the value can be:</p>

<ul>
  <li>time it takes from the code is written to deploy,</li>
  <li>system stability,</li>
  <li>how often bugs are reported compared to earlier, or</li>
  <li>frequency of errors in logs.</li>
</ul>

<p>So, what can you do as a developer on a project that might need some help, you might ask? You can just start by continuously suggesting improvements and showing their value to the customer, or the project manager. Now you just need to keep going and gradually improving the system, one small bit at a time.</p>

<hr />]]></content><author><name>Rustam Mehmandarov</name></author><summary type="html">Common pitfalls in software development and practical suggestions on how to fix them.</summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://mehmandarov.com/assets/images/posts-images/mindthegap.png"/><category term="blog"/><category term="java"/><category term="field notes"/><category term="software development"/><category term="english"/></entry><entry><title type="html">Docker Command Line Survival Guide: The Absolute Basics</title><link href="https://mehmandarov.com/docker-cmd-survival-guide/" rel="alternate" type="text/html" title="Docker Command Line Survival Guide: The Absolute Basics"/><published>2017-07-27T08:23:00+02:00</published><updated>2017-07-27T08:23:00+02:00</updated><id>https://mehmandarov.com/docker-cmd-survival-guide</id><content type="html" xml:base="https://mehmandarov.com/docker-cmd-survival-guide/"><![CDATA[<p><em>A brief introduction to ten essential and absolute basic Docker commands to get you started, and keep you going in the command-line interface.</em></p>

<ul>
  <li><a href="#getting-started">Getting Started</a></li>
  <li><a href="#commands-files-and-folders-inside-a-container">Commands, Files, and Folders Inside a Container</a></li>
  <li><a href="#cleanup">Cleanup</a></li>
</ul>

<hr />

<p>In this post, I decided to share some of the basic commands you might need to get started with Docker. This is neither an extensive list of the commands available, nor all of the commands you might need. This is merely me sharing a prettified list of my cheat sheet for Docker basics with <em>you</em>.</p>

<h2 id="getting-started">Getting Started</h2>

<p>Before we get started, it might be a good idea to note that all of the commands below are written without <code class="language-plaintext highlighter-rouge">sudo</code>. If your installation is not running without <code class="language-plaintext highlighter-rouge">sudo</code> (assuming that you are running Linux), you might want to check out the <a href="https://docs.docker.com/engine/installation/linux/linux-postinstall/" target="_blank">post-installation guide for Linux</a> in the Docker docs.</p>

<h4 id="1-check-if-everything-works">1. Check if Everything Works</h4>

<p>First things first, you can use this simple command to check that your installation is fine. <strong><em>Note:</em></strong> Make sure you have right CPU architecture for your images. Raspberry Pi (ARM) things will not run on x86 architecture, and vice versa.</p>

<p>For x86:</p>

<figure class="highlight"><pre><code class="language-shell_session" data-lang="shell_session"><span class="gp">$</span><span class="w"> </span>docker run docker/whalesay cowsay Hello World!</code></pre></figure>

<p>For Raspberry Pi / AMD:</p>

<figure class="highlight"><pre><code class="language-shell_session" data-lang="shell_session"><span class="gp">$</span><span class="w"> </span>docker run <span class="nt">-d</span> <span class="nt">-p</span> 80:80 hypriot/rpi-busybox-httpd</code></pre></figure>

<p><img src="/assets/images/posts-images/2017-07-27_helloworld-rpi.png" alt="Hello World" class="bigger-image" /></p>

<h4 id="2-list-containers">2. List Containers</h4>

<p>After creating containers, first thing you might want to do is to see what containers you have up and running. To list all running containers you can use:</p>

<figure class="highlight"><pre><code class="language-shell_session" data-lang="shell_session"><span class="gp">$</span><span class="w"> </span>docker ps</code></pre></figure>

<p>This command will give you a list similar to this:</p>

<figure class="highlight"><pre><code class="language-text" data-lang="text">CONTAINER ID        IMAGE                       COMMAND                  CREATED             STATUS                    PORTS                NAMES
e85753d57a67        easypi/dokuwiki-arm         "/bin/sh -c 'php-f..."   1 days ago          Up 23 hours               0.0.0.0:80-&gt;80/tcp   mywiki</code></pre></figure>

<p>However, it will not show you any stopped containers. To list <em>all</em> local containers use the <code class="language-plaintext highlighter-rouge">-a</code> option:</p>

<figure class="highlight"><pre><code class="language-shell_session" data-lang="shell_session"><span class="gp">$</span><span class="w"> </span>docker ps <span class="nt">-a</span></code></pre></figure>

<p>The output will be more like this (note that is shows also stopped, or even failed containers):</p>

<figure class="highlight"><pre><code class="language-text" data-lang="text">CONTAINER ID        IMAGE                       COMMAND                  CREATED             STATUS                    PORTS                NAMES
573193cf1d5e        hypriot/rpi-busybox-httpd   "/bin/busybox http..."   2 days ago          Exited (0) 5 hours ago                         mytest
e85753d57a67        easypi/dokuwiki-arm         "/bin/sh -c 'php-f..."   1 days ago          Up 23 hours               0.0.0.0:80-&gt;80/tcp   mywiki</code></pre></figure>

<p>More on <code class="language-plaintext highlighter-rouge">docker ps</code> in the <a href="https://docs.docker.com/engine/reference/commandline/ps/" target="_blank">Docker docs</a>.</p>

<h4 id="3-list-images">3. List Images</h4>

<p>To list all the images available on your system, simply do this:</p>

<figure class="highlight"><pre><code class="language-shell_session" data-lang="shell_session"><span class="gp">$</span><span class="w"> </span>docker images</code></pre></figure>

<h4 id="4-containers-vs-images">4. Containers vs. Images?</h4>

<p>What is the difference between containers and images, you might wonder? Well, I have a <a href="https://docs.docker.com/engine/userguide/storagedriver/imagesandcontainers/" target="_blank">link</a> for you. This will hopefully help you to understand how Docker manages the data within your images and containers.</p>

<h4 id="5-starting-and-stopping-containers">5. Starting and Stopping Containers</h4>

<p>Another two basic commands &#8211;&#160;<a href="https://docs.docker.com/engine/reference/commandline/start/" target="_blank">starting</a> and <a href="https://docs.docker.com/engine/reference/commandline/stop/" target="_blank">stopping</a> containers:</p>

<figure class="highlight"><pre><code class="language-shell_session" data-lang="shell_session"><span class="gp">$</span><span class="w"> </span>docker start &lt;container_id&gt;
<span class="gp">$</span><span class="w"> </span>docker stop &lt;container_id&gt;</code></pre></figure>

<p><strong><em>Note:</em></strong> The <code class="language-plaintext highlighter-rouge">docker run</code> command first creates a writeable container layer over the specified image, and then starts it using the specified command. That is, <code class="language-plaintext highlighter-rouge">docker run</code> is equivalent to the API&#8217;s <code class="language-plaintext highlighter-rouge">/containers/create</code>, and then <code class="language-plaintext highlighter-rouge">/containers/&lt;id&gt;/start</code>.</p>

<hr />

<h2 id="commands-files-and-folders-inside-a-container">Commands, Files, and Folders Inside a Container</h2>

<h4 id="6-run-any-command-from-a-container">6. Run Any Command from a Container</h4>
<p>You can <a href="https://docs.docker.com/engine/reference/commandline/exec/" target="_blank">run any command</a> in a running container just knowing its ID (or name):</p>

<figure class="highlight"><pre><code class="language-shell_session" data-lang="shell_session"><span class="gp">$</span><span class="w"> </span>docker <span class="nb">exec</span> <span class="nt">-it</span> &lt;container_id_or_name&gt; <span class="nb">echo</span> <span class="s2">"Hello from container!"</span></code></pre></figure>

<h4 id="7-getting-into-containers">7. Getting Into Containers</h4>

<p>Since you can run any command, then you can (obviously) also run a shell from a container; if you have any. This will be a bit similar to running an <code class="language-plaintext highlighter-rouge">ssh</code> command to connect remotely to a regular Linux box (given you have <code class="language-plaintext highlighter-rouge">bash</code> or <code class="language-plaintext highlighter-rouge">sh</code> in the container):</p>

<figure class="highlight"><pre><code class="language-shell_session" data-lang="shell_session"><span class="gp">$</span><span class="w"> </span>docker <span class="nb">exec</span> <span class="nt">-it</span> &lt;container_id_or_name&gt; bash
<span class="gp">$</span><span class="w"> </span><span class="c"># or:</span>
<span class="gp">$</span><span class="w"> </span>docker <span class="nb">exec</span> <span class="nt">-it</span> &lt;container_id_or_name&gt; sh</code></pre></figure>

<h4 id="8-copy-files-from-and-to-containers">8. Copy Files From and To Containers</h4>

<p>Another useful trick you might need is to copy some files to and from a container. Your friend here is the <code class="language-plaintext highlighter-rouge">docker cp</code> command (<a href="https://docs.docker.com/edge/engine/reference/commandline/cp/" target="_blank">link to the docs</a>):</p>

<figure class="highlight"><pre><code class="language-shell_session" data-lang="shell_session"><span class="gp">$</span><span class="w"> </span><span class="c"># To container:</span>
<span class="gp">$</span><span class="w"> </span>docker <span class="nb">cp </span>foo.txt &lt;container_name&gt;:/foo.txt
<span class="gp">$</span><span class="w"> </span><span class="c"># From container:</span>
<span class="gp">$</span><span class="w"> </span>docker <span class="nb">cp</span> &lt;container_name&gt;:/foo.txt foo.txt</code></pre></figure>

<hr />

<h2 id="cleanup">Cleanup</h2>

<p>After playing round with all the images and containers, you might realize that you have quite a collection of these on your drive, just taking up space.</p>

<h4 id="9-remove-containers">9. Remove Containers</h4>

<p>To remove the unused or unwanted containers, you can run the <code class="language-plaintext highlighter-rouge">docker rm</code> command with the IDs of those images. The IDs can be retrieved with the <code class="language-plaintext highlighter-rouge">docker ps -a</code> command, mentioned above.</p>

<figure class="highlight"><pre><code class="language-shell_session" data-lang="shell_session"><span class="gp">$</span><span class="w"> </span>docker <span class="nb">rm</span> &lt;container_id&gt;</code></pre></figure>

<h4 id="10-remove-images">10. Remove Images</h4>

<p>The <code class="language-plaintext highlighter-rouge">docker rmi</code> command followed by the IDs of images will help you to remove the unused or unwanted images. The abovementioned <code class="language-plaintext highlighter-rouge">docker images</code> command will help you finding the correct IDs for the images in question.</p>

<figure class="highlight"><pre><code class="language-shell_session" data-lang="shell_session"><span class="gp">$</span><span class="w"> </span>docker rmi &lt;container_id&gt;</code></pre></figure>

<p><em>Have fun!</em></p>

<hr />]]></content><author><name>Rustam Mehmandarov</name></author><summary type="html">Ten essential Docker commands to get you started with the command-line interface.</summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://mehmandarov.com/assets/images/posts-images/street-art-container_small.jpeg"/><category term="blog"/><category term="docker"/><category term="command line"/><category term="field notes"/><category term="english"/></entry><entry><title type="html">Personal Data Security</title><link href="https://mehmandarov.com/personal-data-security/" rel="alternate" type="text/html" title="Personal Data Security"/><published>2017-07-21T10:23:00+02:00</published><updated>2017-07-21T10:23:00+02:00</updated><id>https://mehmandarov.com/personal-data-security</id><content type="html" xml:base="https://mehmandarov.com/personal-data-security/"><![CDATA[<p><em>Are you sure you are not leaking sensitive data? Or how one of the Norway&#8217;s biggest Apple Premium Resellers and service centers was leaking data to anyone who would bother to click a button.</em></p>

<ul>
  <li><a href="#the-story">The Story</a></li>
  <li><a href="#information-security">Information Security</a></li>
  <li><a href="#norwegian-personal-data-act-personopplysningsloven">Norwegian Personal Data Act (Personopplysningsloven)</a></li>
  <li><a href="#eu-gdpr">EU GDPR</a></li>
  <li><a href="#the-bottom-line">The Bottom Line</a></li>
</ul>

<hr />

<h2 id="the-story">The Story</h2>
<p>It all started when I had to deliver my Apple device for service due to some hardware issues to one of the biggest Apple Premium Resellers and service centers in Norway. After handing in the product, I got an SMS and an email containing a link to a website where I could track the progress of the service online. So far, so good.</p>

<p>While logging in I realized that I already had an account, but did not have the password, so I decided to reset that &#8211; that&#8217;s where it all started.</p>

<p>The first thing I did, was to push the big, blue &#8220;Forgot My Password&#8221; button. Unsure if I had to type in the email first, or if I would be forwarded to another page, where I would have to provide my account details to process with the password reset procedure, I just clicked the button.</p>

<figcaption class="caption">Resetting password</figcaption>
<p><img src="/assets/images/posts-images/2017-07-21_reset_password_1.png" alt="Resetting password" /></p>

<p>However, instead of being redirected to a new page, or getting an error about the missing e-mail in the form, I was presented with this page. Are you noticing anything strange?</p>

<figcaption class="caption">Your password has been reset</figcaption>
<p><img src="/assets/images/posts-images/2017-07-21_reset_password_2.png" alt="Your password has been reset" /></p>

<p>Well, yes, the site reset the password and sent it over to an email and as an SMS. Cool! The only problem was that at that point it could not have any idea who I was, since I have not provided any information about myself yet, and there were no cookies to identify myself to that site.</p>

<p>Another problem there was that the phone number is shown in clear text (hidden here) was not mine. So, I just reset the password and sent it over to some random user &#8211; possibly the first, or the last one in the users table. I tried a few times just to make sure that it was not my fault, and I was still resetting the password for the same person <em>(sorry, total stranger!)</em>.</p>

<p>Having worked with systems development for quite some time, I shrug my shoulders, slightly shook my head, mumbled something about weird bugs and reset my password. This time by providing my e-mail address, proceeding to check the status of my device.</p>

<p>Then, it suddenly hit me. By only providing an email to a service, I could see a confirmation about my password is sent to <em>my mobile phone number</em> registered in the system &#8211; <em>in clear text</em>!</p>

<p>While the first bug (resetting the password for a random person) might be just annoying to a small group of users, the second one (dumping the phone numbers from the database in clear text) was much worse for a bigger group of people. Why might you ask?</p>

<h2 id="information-security">Information Security</h2>
<p>Well, given the fact that the company is being one of the biggest service centers for Apple products, it is very likely to assume that many people would have owned, and sent in for a service an Apple device at some point in the past; thereby getting registered in the service provider&#8217;s database.</p>

<p>So, now I was sitting in front of an unintentional yellow pages (a.k.a. phone directory) service that could provide me with phone numbers of nearly anyone I wanted by just manually typing their emails, or by creating a script that would try scrape the Internet, or just simply construct emails by putting together <code class="language-plaintext highlighter-rouge">firstname.lastname</code> and some <code class="language-plaintext highlighter-rouge">@provider.com</code>, and dumping all the phone numbers from their customer database.</p>

<p>Well, of course, bugs happen, so I don&#8217;t want to jump into conclusions about the lack of proper testing or similar in general.</p>

<p>However, when we provide data to a company, we expect them to handle it with integrity and care, and not leak personal data to the outside world. While phone number might be considered a low-risk data to be leaked for most of us, it might still be quite sensitive for some groups of people, like some high-profile politicians, celebrities, or anybody else how might have a wish, or even a need, to hide their contact information.</p>

<h2 id="norwegian-personal-data-act-personopplysningsloven">Norwegian Personal Data Act (Personopplysningsloven)</h2>
<p>Also, according to The Norwegian Data Protection Authority (Datatilsynet), any information that can be used to identify a person is <a href="https://www.datatilsynet.no/om-personvern/personopplysninger/" target="_blank">considered personal</a>. Further, <a href="http://app.uio.no/ub/ujur/oversatte-lover/data/lov-20000414-031-eng.pdf" target="_blank">Personal Data Act</a> chapter 2, section 13 <em>(Norwegian: [Personopplysningsloven] from 2000)</em> requires that <em>&#8220;the processor shall by means of planned, systematic measures ensure satisfactory data security with regard to confidentiality, integrity, and accessibility in connection with the processing of personal data&#8221;</em>.</p>

<p>Further, according to the Personal Data Act section 46, The Norwegian Data Protection Authority (Datatilsynet) may impose a fee for violations of the act, or the regulations, with an amount up to ten times the <a href="https://www.skatteetaten.no/en/rates/national-insurance-scheme-basic-amount/" target="_blank">basic amount</a> of the National Insurance, equivalent to 925,760 NOK (as of May 2017).</p>

<h2 id="eu-gdpr">EU GDPR</h2>
<p>If the fines mentioned above sound bad, just wait to see how expensive it will get with the introduction of EU General Data Protection Regulation (GDPR) next year.</p>

<p>With the introduction of <a href="http://www.eugdpr.org/" target="_blank">GDPR</a> in 2018, the maximum amount of fines will be raised significantly with an upper limit of 20 million NOK, or the company&#8217;s 4% of the total global annual turnover in the previous fiscal year, if this is higher (GDPR art. 83, item 5).</p>

<h2 id="the-bottom-line">The Bottom Line</h2>
<p>Storing any personal information is an important task and requires rigorous testing and planning on what data you collect, why and how it is protected. Deviating from that can be rather harmful to your company both in regard to reputation and the financial penalties.</p>

<p>That all being said, it is important to note that all of the problems I reported to the company in question were fixed within a few hours. However, I don&#8217;t know for how long that data was available online, and if anyone had taken advantage of the vulnerability of the system.</p>

<p>Least but not last, I would like to thank the company, and especially the company&#8217;s CIO for great communication and quick responses and fast bug fixes.</p>

<hr />]]></content><author><name>Rustam Mehmandarov</name></author><summary type="html">How one of Norway&apos;s biggest Apple service centers was leaking personal data to anyone who would click a button.</summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://mehmandarov.com/assets/images/posts-images/caution_keyboard_small.jpeg"/><category term="blog"/><category term="security"/><category term="data privacy"/><category term="gdpr"/><category term="english"/></entry></feed>

