• Introduction
• Why Versioning?
• Show Me The CODE!
• 1. URL Versioning
• 2. Header Versioning
• 3. Media Type Versioning
• 4. Request Parameter Versioning
• 5. Bonus: Combining Strategies
• 6. End-Point Deprecation
• Summary Comparison
• Conclusion
• What’s Next?

Introduction

When working with APIs over time we would often need to make some changes to end-point definitions – like adding or deleting resources or changing the attributes for a resource. To ensure backwards compatibility, we often have to introduce versioning for our APIs. APIs, like all software, evolve. You might be adding optional fields or introducing a breaking change. At some point, you will need versioning to support coexistence of the old and new consumers.

However, versioning the API endpoint introduces a question of how this should be done. In this post, we’ll explore several common API versioning strategies, using Jakarta EE and Java.

💡 Note: There is no silver bullet – instead, we’ll explore pros, cons, and real-world fit.

Why Versioning?

Why not just change the API?
Because breaking contracts is dangerous — clients may not update in sync, and you’ll break production consumers.

Versioning allows you to:

• Support legacy clients
• Introduce new features safely
• Deprecate responsibly

⚠️ Caution: Versioning can cause “version explosion.” Each version increases long-term maintenance cost – aka technical debt.

Best Practice: Prefer backward-compatible changes (e.g., adding fields) whenever possible. To mitigate risks, it’s important to follow best practices for versioning and provide clear documentation and migration paths for users. Also, remember to deprecate old versions to minimize maintenance efforts.

Show me the CODE!

I have created a repository called Random Strings to showcase various concepts. For this blogpost, I would recommend having a look at RandomStringsAPIDemoController.java and request_examples.http. You will find all the info on building and running the code in the repo’s README.md file. Each section below will contain “How to call it” part with an example using curl or HTTP-files, and will be based on this repo.

1. URL Versioning

What it looks like

A version appears directly in the URI path. If your API is at https://example.com/api, and the current version is version 1, the URL for a resource might look like this: https://example.com/api/v1/resource:

@GET
@Path("/v2/")
@Produces(MediaType.APPLICATION_JSON)
public Response getV2() {
    return Response.status(Response.Status.NOT_IMPLEMENTED)
        .entity("This v2 using *path versioning* of the API is not implemented.")
        .build();
}

How to call it

cURL:

curl -X GET http://localhost:8080/api/rnd/v2/ \
  -H "Accept: application/json"

HTTP Request (.http file):

GET http://localhost:8080/api/rnd/v2/
Accept: application/json

✅ Pros:

• Simple and intuitive. Visible.
• Easy to test (e.g., with curl or Postman directly in a browser).
• Plays well with gateways and reverse proxies.
• Clear visual distinction between versions.

❌ Cons:

• Pollutes the URI with versioning logic.
• Breaks REST’s principle of stable resource identifiers.
• Clients have to update URLs when migrating.
• Risk of accumulating too many legacy versions.
• Can result in cluttered and difficult-to-read URLs if there are multiple versions of the API.

🔍 However: Despite its REST purism flaw, URL versioning is extremely practical and widely adopted.

2. Header Versioning

What it looks like

Client specifies version in a custom HTTP header (e.g., Accept-Version, X-API-Version, etc.):

@Path("/hi2")
@GET
@Produces({"application/json"})
public String entryPoint2(@HeaderParam("Accept-Version") String apiVersion) {
    if (apiVersion == null || apiVersion.isEmpty()) {
        return "Default unversioned endpoint hit.";
    }
    String message = "Versioned: Using custom headers. Using version: " + apiVersion +".";
    return message;
}

How to call it

Note: This is for demo purposes only. It has to have a different URL than the regular API; otherwise, it will also intercept calls that do not contain the Accept-Version header.

cURL:

curl -X GET http://localhost:8080/api/rnd/versioned/ \
  -H "Accept: application/json" \
  -H "Accept-Version: 2"

HTTP Request (.http file):

GET http://localhost:8080/api/rnd/versioned/
Accept: application/json
Accept-Version: 2

✅ Pros:

• Keeps URL structure clean and predictable.
• Closer to HTTP semantics (headers = metadata).
• Allows centralized versioning logic in filters/interceptors.

❌ Cons:

• Not self-descriptive — clients must “know the secret handshake”.
• Poor discoverability (not visible in browser without tools).
• Breaks caching in some proxies/CDNs unless explicitly configured.
• Adds complexity to tooling and testing.

⚠️ Challenge: Header versioning can feel “invisible” and cause developer confusion if not well-documented.

3. Media Type Versioning (Content Negotiation)

What it looks like

Client specifies version via a custom media type in the Accept header. This is sometimes called Content Negotiation versioning.

Accept: application/hi.v3+json

In Jakarta EE:

@Path("/hi")
@GET
@Produces({"application/hi.v3+json", "application/hi.v4+json"})
public String entryPoint() throws URISyntaxException {
    String message = "Versioned: Hai there!";
    return message;
}

How to call it

You can request different versions (e.g., v3, v4, v5) by updating the media type:

cURL:

curl -X GET http://localhost:8080/api/rnd/ \
  -H "Accept: application/rnd.v3+json"

HTTP Request (.http file):

GET http://localhost:8080/api/rnd/
Accept: application/rnd.v3+json

✅ Pros:

• Very REST-compliant: changes representation, not resource.
• URI remains stable.
• Supports richer format negotiation (e.g., XML, HAL, etc.).

❌ Cons:

• Requires strict control over media types.
• Not all clients/tooling handle custom media types well.
• Breaks with some reverse proxies and middleware that don’t forward full Accept headers.
• More work to configure content negotiation.

🧪 Observation: Elegant in design, but rarely used consistently in real-world public APIs.

4. Request Parameter Versioning

What it looks like

Technically, it is also possible for the client to specify the version in a URL query parameter (e.g., ?version=2). This, however, might not be a suggested strategy, in my opinion.

https://example.com/api/resource?version=2

How to call it

cURL:

curl -X GET http://localhost:8080/api/rnd?version=2 \
  -H "Accept: application/json"

HTTP Request (.http file):

GET http://localhost:8080/api/rnd?version=2
Accept: application/json

✅ Pros:

• Simplicity & discoverability: Easy to test in a browser without specialized tools.
• Defaulting logic: Straightforward to implement “default to latest” if the parameter is omitted.
• Caching friendly: CDNs treat different query params as unique resources by default.

❌ Cons:

• URI Pollution: Mixes resource identification with technical metadata.
• Routing complexity: Routing based on query parameters usually requires custom middleware or manual logic inside the controller.
• Harder to generate clean, automated documentation (like OpenAPI) when multiple versions share the same path.

5. Bonus: Combining Strategies - Transparent URI Rewriting (Enterprise Pattern)

In large enterprises, you might find that different clients have different needs. Some prefer the explicitness of URL versioning, while others require the clean URIs of Header versioning. You don’t have to choose just one—you can support both without duplicating your backend routing logic.

The common practice is to structure all your resource classes using URL versioning (e.g., @Path("/v1/resource")), but use a @PreMatching Filter to intercept requests and transparently rewrite the URI if a client uses a header instead.

Here is what that looks like in Jakarta EE using a ContainerRequestFilter:

@Provider
@PreMatching
public class HeaderVersionFilter implements ContainerRequestFilter {

      @Override
      public void filter(ContainerRequestContext ctx) {
          String path = ctx.getUriInfo().getPath();

          // If the path is already versioned (e.g., starts with v1, v2), let it pass
          if (path.matches("v\\d+(/.*)?")) return;

          // Otherwise, check if the client provided a version header
          String version = ctx.getHeaderString("X-API-Version");

          if (version != null && !version.isEmpty()) {
              // Transparently rewrite the URI internally to match our URL-based routes
              String newPath = "v" + version + "/" + path;
              URI baseUri = ctx.getUriInfo().getBaseUri();
              URI newUri = UriBuilder.fromUri(baseUri).path(newPath).build();

              ctx.setRequestUri(baseUri, newUri);
          }
      }
}

✅ Pros:

• Ultimate Flexibility: Clients can use http://api.example.com/v2/resource OR http://api.example.com/resource with an X-API-Version: 2 header.
• Single Source of Truth: Your backend controllers only need to use @Path("/v2/"). You don’t have to write duplicate methods to handle both headers and paths.

❌ Cons:

• Magic Routing: It introduces a layer of “magic” where the requested URI differs from the routed URI, which can briefly confuse new developers debugging the application.

💡 Want to know more? Read up on terms Version Normalization and Internal Decoupling.

6. End-Point Deprecation

Eventually, you will need to retire old API versions. Remember: every old version you keep around is technical debt — it increases long-term maintenance cost. When deprecating an endpoint, consider the following best practices:

1. Update the Docs: Use OpenAPI’s @Operation annotation to clearly mark it as deprecated.
2. Add @Deprecated: Use the Java @Deprecated annotation where necessary.
3. HTTP Redirects: Consider returning HTTP codes like 302 Found or 301 Moved Permanently after some time.
4. Add a Link header: Provide a link to the new version in the response headers.
5. Log / Count calls: Track usage (e.g., with MicroProfile @Counted) to know when it is safe to finally remove the endpoint.

Here is a practical example in Jakarta EE showing how to deprecate an endpoint, add a Link header, and track metrics:

@GET
@Path("v0.1/")
@Produces(MediaType.APPLICATION_JSON)
@Operation(summary = "DEPRECATED. Use v2 now. Returns the adjective-noun pair",
           description = "Deprecated function. The pair of one random adjective and one random noun is returned as an array.")
@Counted(name = "totalCountToRandomPairCalls_Versioned_Path_DEPRECATED",
         absolute = true,
         description = "Deprecated function call: Total number of calls to random string pairs.",
         tags = {"calls=pairs"})
@Deprecated
public Response getRndStringPathDeprecated() {
    URI newVersionURI = UriBuilder.fromUri("/api/rnd/v2/").build();
    Link newVersionLink = Link.fromUri(newVersionURI).rel("alternate").build();
    return Response.ok("Deprecated response", MediaType.APPLICATION_JSON)
            .header(jakarta.ws.rs.core.HttpHeaders.LINK, newVersionLink.toString())
            .header("X-API-Version", "0.1")
            .build();
}

How to call it (Deprecated endpoint)

cURL:

curl -X GET http://localhost:8080/api/rnd/v0.1/ \
  -H "Accept: application/json"

HTTP Request (.http file):

GET http://localhost:8080/api/rnd/v0.1/
Accept: application/json

Summary Comparison

The following table summarizes all the different routing strategies implemented in the demo project, illustrating how the HTTP method, path, and headers combine to invoke the correct Java method. The method names refer to the methods in RandomStringsAPIDemoController.java (or RandomStringsController.java):

Conclusion

There is no single correct approach to API versioning. For most teams and public APIs, URL versioning is good enough—it’s visible, easy to test, and plays well with existing tooling. However, you might want to use header versioning if your APIs are primarily consumed by internal services or SDKs that can abstract away the complexity. Reserve media type versioning for hypermedia-rich or REST-purist APIs, and only if your tooling supports it end-to-end.

Consider who your consumers are, whether your API is public or internal, your infrastructure maturity, and your team’s ability to support multiple versions.

What’s Next?

Versioning is just one part of building robust REST APIs. If you want to dive deeper, have a look at the API Guide for Java repository and the slides in the presentation folder. They cover documentation with OpenAPI, security best practices (like RBAC and JWT integration), advanced patterns (pagination, async APIs), and going beyond REST with gRPC and GraphQL.

Happy coding!

• Introduction
• Setup
• Conclusion

Introduction

The built-in development mode for Quarkus is a great functionality that lets you update the application code, resources, and configurations. Setting it up is a great way to develop your applications locally, as you can immediately see the changes reflected in your application.

Furthermore, we have a remote development mode, which lets you make changes to local files immediately available in a containerized environment. Remote development mode works excellently if the container runs in a local Docker or remote containerized environment.

However, running several simultaneous containers with the remote development mode on, mapped to the same domain, may result in warnings and erratic behavior from the client side.

Setup

Imagine a setup where you are running a set of containers, for example, using docker-compose and mapping them all to my.cluster.host.com (or even localhost)through several ports:

First, you will need to update quarkus.live-reload.url in the properties for all the apps (see docs on where and how to do this). Update the settings to the correct domain and port (in our case, it is 8080, 8081, or 8082):

quarkus.live-reload.url=http://localhost:8081

Next, try starting your containers with the remote development mode enabled and connect to the application from a terminal or an IDE. For the second and the consecutive applications, the attempts to establish a connection you will see the following message in the logs:

$> ./mvnw quarkus:remote-dev -Dquarkus.profile=dev

< ... >

[WARNING] Changed debug port to 57409 because of a port conflict
Listening for transport dt_socket at address: 57409

< ... >

Note: Fallback ports will be random and may vary from the one above.

This setup will break the remote reloading from the terminal on the client side (i.e., your IDE). Two or more of your client applications now see that the default port 50005 for a remote debug is in use and start with a new, random port.

The simple fix is to update the debug ports for all other applications to something other than 5005, such as 6006 and 6007. Custom debug ports can be set in the pom.xml files, under quarkus-maven-plugin, for each of the applications that require this update:

<build>
  <plugins>
    <plugin>
      <groupId>io.quarkus.platform</groupId>
      <artifactId>quarkus-maven-plugin</artifactId>
      <version>${quarkus.platform.version}</version>
      <!-- ADD THE CONFIGURATION MENTIONED BELOW THIS LINE -->
      <configuration>
        <debug>6006</debug>
      </configuration>
...

You can choose whether to update the debug ports for all applications in the cluster or for all applications except one, which will get the default port.

Now, you will need to rebuild your apps and re-initiate the remote development mode for each container. And, voilà, everything works!

One last note: Please ensure you do not use the remote development functionality in the production environment.

Conclusion

A tiny config update brings back the development joy of using remote development mode for more than one container simultaneously.

Happy coding!

• Introduction
• Principles
• Conclusion

Introduction

In the first part, we have summed up all the essential elements to consider when working with Cloud and securing cloud-native applications/platforms. In this post, we would like to give you some concrete principles and tips for creating more secure applications.

Principles

Multi-Layered Defense

Keywords: general

First of all, a more generic but important principle: It would be best to look at security as a whole – integrating various security layers on multiple levels in any system. It should include cyber-security plans for:

1. Devices
2. Applications
3. Networks
4. Infrastructure
5. People

Think of this principle as all the layers of clothing you wear to protect yourself from cold and bad weather. If one of the layers is compromised, there is always another to keep you warm and dry.

Identity and Access Management (IAM) Misconfiguration

Keywords: network, permissions

You need to control access and permissions meticulously and over time. Things to consider:

• Implementing role-based access control (RBAC)
• Principle of least privilege
• Routines for updating and removing permissions when they are no longer needed.
• Explore possibilities for using time-based conditions for IAM policies.

API Security

Keywords: endpoints, permissions

• APIs act as the gateways to your application and data. Securing access to and securing them from known vulnerabilities is paramount to prevent unauthorized access and data breaches.
• Utilize authentication, authorization, and API gateways to control access and protect sensitive information. Don’t forget to monitor the software or libraries that make APIs available (e.g., runtimes, middleware)

Data Encryption

Keywords: data

• Safeguarding data at rest, in transit, and during processing is critical for your applications.
• Utilize encryption, tokenization, and data masking techniques to ensure data protection. Removing unnecessary sensitive information can simplify some of these tasks.
• If a platform or a Cloud provider provides the encryption, consider if you would like to use the standard keys for encryption or “bring your own” and manage them yourself or through a third party.
• Beware: Don’t write your own crypto! Ever.

Zero Trust

Keywords: network, permissions

• The Zero Trust security model assumes that no one is inherently trustworthy, even those within your network.
• This is opposed to more traditional approaches where perimeter security was prioritized over security inside the network.
• Adopting this approach, every request, user, and device is thoroughly verified before gaining access.
• Again: Implement the principle of least privilege, where users are only granted the minimum level of access required to perform their tasks.

Software Supply Chain Security

Keywords: software, environment

• Create Software Bill of Materials (SBOM) for your software
• Governance: Know where all the building blocks (artifacts) of your software are coming from.
• Automate security checks within your CI/CD pipeline to catch vulnerabilities early and often.
• Use static code analysis with tools like SonarQube to scan your code for potential security flaws and integrate those checks into your CI/CD pipeline to ensure continuous security monitoring.
• Use tools to monitor not only the code you develop yourself but also all the third-party libraries you utilize in your code.
• With DevSecOps, automated security security is becoming integral to the development process. Adopt it if you haven’t done so already.

Secure Containerization

Keywords: software, environment

• Containerization and orchestration technologies, like Docker and Kubernetes, offer exceptional flexibility but also introduce security concerns.
• Securing containers and managing their lifecycle is vital to ensure a safe cloud environment.
• For example, use container scanning tools to identify vulnerabilities within container images before deploying them.
• Additionally, enforce strict security policies and segregate workloads using Kubernetes namespaces.

Continuous Monitoring and Incident Response

Keywords: software, environment

• The cloud landscape is constantly changing, and threats evolve rapidly. This means that we need to monitor not only for known threats but also for anomalies.
• Continuous monitoring and proactive incident response are essential to detect anomalies and respond swiftly to security incidents.
• For example, use cloud-native monitoring tools your Cloud or platform provider provides.
• Have good logging, but remember that more is not always better – log relevant information.

Human Factors (including Social Engineering, Misconfigurations, and Human Errors)

Keywords: people, human factors

• 82% of incidents are caused by human factors (2022 Data Breach Investigations Report)
• Creating secure applications also implies providing security training for the system users.
• Social engineering and human factor has proven to be essential to creating secure applications.
• Consider running security awareness campaigns and employee training from user and developer perspectives.
• Automate routine and mundane tasks – humans often don’t enjoy carrying out tasks like this and are prone to errors; computers, on the other hand, excel at tasks like this!

Conclusion

You have probably heard that nothing is stronger than its weakest link. Therefore, it is important to look at various sides of the security. Especially in the Cloud, one size does not fit all when it comes to security. Cloud platforms, software, and threats constantly evolve and add to the complexity of creating secure applications.

Here, we have seen some of the principles to consider regarding the security of the platforms and application development for the Cloud and cloud-native applications in general.

Finally, note that this is not an exhaustive list but is instead meant to serve as a stepping stone to more secure application development.

• Introduction
• Key Elements of a Cloud Security Architecture
• Responsibilities
• Constantly Evolving Landscape
• Platform Security Architecture
• Application Security Architecture
• Conclusion

Introduction

Whether you are running on the Cloud or not it is all about the CIA triad model – Confidentiality, Integrity, and Availability.

When thinking about Cloud Security Architecture we need to be able to think about the whole stack. Of course, we don’t need to think about all the moving parts alone – it is a shared responsibility between the Cloud service provider and you, the user of the platform.

Key Elements of a Cloud Security Architecture

Let’s first start by defining the key elements of a Cloud Security Architecture, divided across the layers of the stack, based on the Cloud Security Alliance (CSA) stack model.

Now, we can also mention some of the main challenges related to security, divided into separate groups, and try to map them to the CIA triad model that we have mentioned earlier.

Network and Storage

• Data Encryption
• Network Security

Application layer

• Application Security
• Logging and Monitoring
• Identity and Access Management (IAM)

Observability, and traceability

• Incident Response and Recovery
• Vendor and Third-Party Risk Management

DevOps

• Automation and Orchestration
• Resilience and High Availability

General

• Compliance and Governance
• User Training and Awareness
• Cloud Provider Security Features

Responsibilities

Shared Responsibility + Intersection of Responsibilities

Addressing all these challenges is a shared responsibility between the Cloud service provider and the customer and the division will vary depending on the type of the solution and whether you are using IaaS, PaaS, or SaaS.

Typically, Cloud service providers will take care of the lower parts of the stack, like physical, infrastructure, and platform security, while customers will be responsible for creating secure applications, securing their data, creating proper Identity and Access Management (IAM), and configuration management.

An effective overlap and a clear understanding of the responsibilities ensure comprehensive security coverage across all layers.

Constantly Evolving Landscape

Evolving Landscape == Constant Change

One of the differentiating factors from regular application development is the constant change and evolution of the platform and tooling on one side, and the constantly evolving types of attacks and possibly larger attack surfaces on the other side.

These factors will lead to changes in the model and the responsibility division. The same might be influenced by the new services being introduced both from the side of the Cloud service provider and the customer (app developer).

Therefore, regular communication between the parties involved and staying updated on their security practices is essential to ensure secure Cloud applications.

Types of the Cloud Security Architecture

The Cloud Security Architecture is twofold – you will need to choose a platform for running your application and think about the security of the application you will be deploying on that platform.

Platform Security Architecture

Let’s start with defining the types of platforms and list some of the key elements to consider when choosing a platform type.

Public Cloud Security Architecture

• Designed for cloud services provided by third-party vendors (e.g., AWS, Azure, Google Cloud).
• Focuses on securing data and applications hosted on shared infrastructure.
• Utilizes the security features provided by the cloud service provider (CSP) while also implementing * additional security measures.
• Emphasizes network segmentation, encryption, IAM, and monitoring.

Private Cloud Security Architecture

• Created for cloud environments dedicated to a single organization.
• Offers more control over security settings and configurations.
• Often used by organizations with strict compliance requirements or sensitive data.
• Implements strong access controls, encryption, and strict network isolation.

Hybrid Cloud Security Architecture

• Combines public and private clouds to take advantage of the benefits of both deployment models
• Security architecture addresses integration challenges and ensures consistency across environments
• Emphasizes secure communication between on-premises and cloud components
• Requires seamless identity and access management across both environments

Multi-Cloud Security Architecture

• Involves using services from multiple cloud providers simultaneously
• Ensures compatibility and security across diverse cloud platforms
• Requires careful management of authentication, authorization, data protection, and compliance measures
• Aims to prevent vendor lock-in and distribute risk

Application Security Architecture

Here are some things you will need to think about when developing modern applications for the Cloud and the cloud-native world.

1. Secure Your Code

• Software Supply Chain Security: Securing and monitoring your artifacts and third-party libraries.
• Making sure the code you have written is secure: OWASP Top 10, static code analysis, coding best practices.

2. Your Container (and Serverless) Security Architecture

• Specifically addresses security for containerized applications (e.g., Containers, Kubernetes) and serverless computing (e.g., AWS Lambda, Azure Functions, Cloud Functions, or Cloud Run on Google Cloud)
• Focus on securing microservices, communication between them, their orchestrators, and function-as-a-service (FaaS) platforms
• Involves isolating containers, securing images, and managing runtime security

3. Add DevSecOps Architecture Practices

• Integrate security practices into the DevOps process: DevSecOps
• Ensure security is considered at every stage of application development and deployment
• Involves automated security testing, vulnerability scanning, and security policy enforcement

4. Cross-application and Cross-container communication: Zero Trust Security Architecture

• Assume no trust by default and require strict authentication and authorization for all users and devices
• Focus on identity verification, principle of least privilege, and continuous monitoring
• Suitable for cloud environments where traditional perimeter defenses are less effective

5. Physical security: Edge Cloud Security Architecture

• Address security concerns at the edge of the network, closer to where data is generated and consumed
• In case of having local edge hardware devices consider also physical security of those devices
• Involves considerations like local processing, secure communication, and protection against threats targeting edge devices

6. Compliance-Centric Security Architecture

• Tailored to meet specific regulatory compliance requirements (e.g., GDPR, HIPAA, PCI DSS)
• Focus on implementing controls and safeguards to adhere to relevant standards

Conclusion

We have seen the key elements of the cloud security architecture and the building blocks of the whole stack. Furthermore, we have looked at the various types and elements to consider when it comes to the security of the platforms and application development. This is a stepping stone to categorize and group some of the main things you will need to consider when working with the Cloud and cloud-native applications.

** Illustrations in this post: Rustam Mehmandarov.

Welcome back!

Finally, after a long hibernation, this site is returning to life. Watch out for new posts coming to servers near you!

I look forward to seeing how the site will develop and future posts. Stay tuned!

• Step 1: Define Pipeline Options
• Step 2: Create the Pipeline
• Step 3: Apply Transformations
• Step 4: Run it!
• Conclusion

In this post, I would like to show you how you can get started with Apache Beam and build the first, simple data pipeline in 4 steps.

Step 1: Define Pipeline Options

Let’s start with creating a helper object to configure our pipelines. This is not an absolute necessity, however defining the pipeline options might save you some time later, especially if your pipeline is dependent on a few arguments, that might have pre-defined, default values that you don’t want to provide at every run.

public interface OsloCityBikeOptions extends PipelineOptions {

    /**
     * By default, the code reads from a public dataset containing a subset of
     * bike station metadata for city bikes. Set this option to choose a different input file or glob
     * (i.e. partial names with *, like "*-stations.txt").
     */
    @Description("Path of the file with the availability data")
    @Default.String("src/main/resources/bikedata-stations-example.txt")
    String getStationMetadataInputFile();
    void setStationMetadataInputFile(String value);

    // some other options here...
}

Step 2: Create the Pipeline

Now that you have created the pipeline options object, you will need to create the pipeline object itself and provide the options to it:

OsloCityBikeOptions options = 
        PipelineOptionsFactory.fromArgs(args)
                                .withValidation()
                                .as(OsloCityBikeOptions.class);

Pipeline pipeline = Pipeline.create(options);

(Check out the documentation for the PipelineOptionsFactory class for the description of the methods used above.)

Step 3: Apply Transformations

After defining the pipeline and providing the options class, we can start by applying the transformations using .apply(...). Those can be chained after each other by applying yet another .apply(...), for instance:

PCollection <KV<Integer, LinkedHashMap>> stationMetadata = pipeline
                .apply("ReadLines: StationMetadataInputFiles", TextIO.read().from(options.getStationMetadataInputFile()))
                .apply("Station Metadata", ParDo.of(fnExtractStationMetaDataFromJSON()));
                .apply(MapElements.into(TypeDescriptor.of(String.class)).via(o -> o.toString()))
                .apply("WriteStationMetaData", TextIO.write().to(options.getMetadataOutput()));

Note that a PCollection<T> is an immutable collection of values of type T and that you can provide names for the transformations as the first string argument in the apply(), like in the first two and the last apply methods.

Here we can also specify custom transformations that can be done in parallel. In Beam, they are being referred to as ParDo methods. They are similar to the Mapper or Reducer class of a MapReduce-style algorithm. In this post, we will not be focusing on the contents of such pipeline (i.e. what it is doing), but a simple example of a ParDo can be looking like the second apply in the code above (look for the link in the conclusion for the entire running example).

pipeline.apply("Station Metadata", ParDo.of(fnExtractStationMetaDataFromJSON()));

Step 4: Run it!

After defining the pipeline, its options, and how they are connected, we can finally run the pipeline. The great thing about running the pipelines in Apache Beam is that it is very easy to switch between various runners. Beam provides a portable API layer for building sophisticated pipelines that may be executed across various execution engines or runners. In our example, we can switch from running the pipeline locally (with direct-runner), to running the same pipeline in the Cloud as a managed service (with dataflow-runner) by simply adjusting the values we provide when running the code.

Local runner

Here is an example of running the pipeline with direct-runner:

mvn compile exec:java \
      -Pdirect-runner \
      -Dexec.mainClass=com.mehmandarov.beam.OsloCityBike \
      -Dexec.args="--inputFile=src/data-example.txt \
      --output=bikedatalocal"

Dataflow runner

And here is the example of running the same pipeline in the Cloud as a managed service, using Google Cloud Dataflow. Note that most of the parameters provided are still the same, with a few additional parameters needed for this specific runner.

mvn compile exec:java \
      -Pdataflow-runner \
      -Dexec.mainClass=com.mehmandarov.beam.OsloCityBike \
      -Dexec.args="--project=rm-cx-211107 \
      --inputFile=gs://my_oslo_bike_data/data-2018-*.txt \
      --stagingLocation=gs://my_oslo_bike_data/testing \
      --output=gs://my_oslo_bike_data/testing/output \
      --tempLocation=gs://my_oslo_bike_data/testing/ \
      --runner=DataflowRunner \
      --region=europe-west1"

Other runners

In case you would like to be using various runners or interested in switching between them, it might be a good idea to check the capability matrix in the documentation, as the core concepts of Beam Model can sometimes be implemented to varying degrees in each of the Beam runners.

Conclusion

We have now seen the basic steps needed to create a simple data-parallel processing pipeline and how that can be run and deployed both in the local and managed Cloud environments. We are were also able to run the same pipeline with just a few adjustments to the command line parameters and, in our case, without any changes to the pipeline code.

The entire working example that we have been using here can be found in my GitHub repository, as well as a more advanced example in another repository.

• Intro
• TL;DR: Getting Graph Representation
• A Full Example
• What Now?

Intro

Constructing advanced pipelines, or trying to wrap your head around the existing pipelines, in Apache Beam can sometimes be challenging. We have seen some nice visual representations of the pipelines in the managed Cloud versions of this software, but figuring out how to get a graph representation of the pipeline required a little bit of research. Here is how it is done in a few steps using Beam’s Java SDK.

TL;DR: Getting Graph Representation

If you just want to see a few lines that let you generate the DOT representation of the graph, here it is:

import org.apache.beam.runners.core.construction.renderer.PipelineDotRenderer;

Pipeline p = Pipeline.create(options);
// do stuff with your pipeline
String dotString = PipelineDotRenderer.toDotString(p);

Now, if you want a slightly more comprehensive example, keep on reading.

A Full Example

Here we will be using word count example, particularly the MinimalWordCount class.

Adding Maven Dependency

First, we need to add a dependency to the Maven file under <dependencies> section:

<dependencies>
    <!-- ... all the other dependencies you may have -->
    <dependency>
        <groupId>org.apache.beam</groupId>
        <artifactId>beam-runners-core-construction-java</artifactId>
        <version>${beam.version}</version>
    </dependency>
</dependencies>

The Code

Now, we will need to add a few imports (assuming you already added the Maven dependency mentioned earlier):

import org.apache.beam.runners.core.construction.renderer.PipelineDotRenderer;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

To get the DOT representation of the pipeline graph we will be passing the pipeline object to the PipelineDotRenderer class, and in this example, we are only logging the output to the console (hence the log4j imports).

// Create the Pipeline object with the options we defined above
Pipeline p = Pipeline.create(options);

// ... do stuff with your pipeline ...

// Add this piece of code just before running the pipeline:
final Logger log = LoggerFactory.getLogger(MinimalWordCount.class);
String dotString = PipelineDotRenderer.toDotString(p);
log.info("MY GRAPH REPR: " + dotString);

p.run().waitUntilFinish();

That’s it. To see the code in action, run it from the command line:

$ mvn compile exec:java \
        -Dexec.mainClass=org.apache.beam.examples.MinimalWordCount \
        -Pdirect-runner

This code will produce a DOT representation of the pipeline and log it to the console.

A Complete Example

A fully working example can be found in my repository, based on MinimalWordCount code. There, in addition to logging to the console, we will be storing the DOT representation to a file.

In the next section, we will have a brief look at what can be done with the DOT representations.

What Now?

Now that we have a DOT representation of the pipeline graph, we can use it to get a better understanding of the pipeline. For instance, you can generate an SVG or a PNG image from the data. Note that the generated graph might be a bit verbose, but gives a good overview of the pipeline graph.

Here, I have also included examples of the DOT graph and the PNG file generated for that particular pipeline.

Assuming that you have Graphviz tools installed, you can convert a DOT file to a PNG image using this command:

$ dot -Tpng -o pipeline_graph.png pipeline_graph.dot

In addition to Grapgviz (Wikipedia link), there are also online services for converting DOT graphs to graphical representations, like this one.

Update (April 2026): This post has been refreshed to use the modern jakarta.* namespace introduced in Jakarta EE 9+. If you are working with an older codebase that still uses the legacy javax.* namespace, the concepts described here apply identically — only the package prefix differs.

• Intro
• Getting started
• Defining End-Points
  • A (Regular) JSON End-Point
  • An End-Point Returning an Image
  • An End-Point Returning a PDF
• What’s Next?

Intro

In my previous two posts, I have been describing parts of a system for “checking-in” to a location using QR codes in Java. We started with generating QR codes, followed by generating PDF files.

Now, I would like to focus on building microservices around that functionality. We will be creating a few HTTP end-points built with MicroProfile. I will be using the next few posts as an opportunity to show off some of the features that you will be getting out of the box or with minimal effort using MicroProfile.

Getting started

(Assuming that you have Git, Java 9+, and Maven installed.)

Since we will continue using the QR code generator project to showcase various features of MicroProfile, it might be a good idea to familiarize yourself with the code. You might want to start with taking a look at my previous two posts that explain the code for generating QR codes, and generating PDF files in detail.

Now you can clone the project and examine the Maven dependencies in the pom.xml file, as well as any other MicroProfile related dependencies.

After cloning and opening the project in your favorite IDE, build it (again, assuming that you have Java and Maven installed) with the following command in a terminal:

$ mvn clean package

When the project is done building and you have got a Build Success from Maven, you can run the project to make sure everything runs fine:

$ java -jar target/qrcreator.jar

In a few seconds it takes for the app server to start-up, you should be able to access the starter page at http://localhost:8181/qrcreator/index.html.

Defining End-Points

One of the most obvious features any microservice needs is an end-point to receive requests and respond with some kind of data back. Let’s have a closer look into how this is done in MicroProfile. First of all, we will need to define the application path that serves as the base URI for all resource URIs (think of it as a “root” URL) and make sure that the class where it is defined extends jakarta.ws.rs.core.Application, like in ApplicationEntryPoint class here:

package com.mehmandarov.qrcreator;

import jakarta.ws.rs.ApplicationPath;
import jakarta.ws.rs.core.Application;

/**
 * Sets the application path that serves as the base URI for
 * all resource URIs provided by @Path annotation.
 */
@ApplicationPath("/api")

public class ApplicationEntryPoint extends Application {
}

This will set up all the end-point URLs to start with /api, in our case http://localhost:8181/qrcreator/api/.

A (Regular) JSON End-Point

Now, let’s define some endpoints. We will start with a most regular kind – a JSON end-point. This is probably the most common end-point you will encounter.

We will start with creating one that would respond to requests sent to /api/somestring/json. Note that as you can see from the code below, @Path defines somestring in the URL as an id that is passed on as an argument to the createIdKeyTuple method.

We will also define a type of a request (in this case it is a GET request) and specify that it will be returning a JSON document with @Produces annotation.

@GET
@Path("{id}/json")
@Produces("application/json")
public Response createIdKeyTuple(@PathParam("id") String id) {
    ...
}

Now, that we are done with annotations for the method, let’s have a look at the code for this method, that defines a JSON end-point in the QRController class:

@GET
@Path("{id}/json")
@Produces("application/json")
public Response createIdKeyTuple(@PathParam("id") String id) {
    String nameKeyTuple = null;
    try {
        nameKeyTuple = qrCodeContentsSupplier.getQRCodeContents(id);
        return Response.ok(nameKeyTuple).build();
    } catch (InvalidKeySpecException e) {
        e.printStackTrace();
    } catch (NoSuchAlgorithmException e) {
        e.printStackTrace();
    } catch (Exception e) {
        e.printStackTrace();
    }
    return Response.serverError().build();
}

Note that in this code we have only simple exception handling that makes sure we return a correct HTTP status code – OK (200) on a success and server error (500) on an internal error. You can later add other HTTP codes based on your needs.

An End-Point Returning an Image

Returning an image instead of a JSON document is quite similar to the code we have already seen. Here we will have to pay attention to three aspects:

• Different MIME type defined in @Produces annotation: @Produces("image/png")
• Additional elements in the response header that let you control how the created file is displayed in the browser, i.e. shown in the browser “inline”, or made available through a download dialog – "Content-Disposition", "inline;" (see specs for Content-Disposition for more details)
• Additional elements in the response header that let you control the name for the created file: filename=\"" + id + ".png\"

Let’s have a look at the whole method:

@GET
@Path("{id}")
@Produces("image/png")
public Response createQR(@PathParam("id") String id) {
    try {
        byte[] imageData = qrCodeSupplier.qrCodeGenerator(id);
        return Response.ok(imageData)
            .header("Content-Disposition", "inline; filename=\"" + id + ".png\"")
            .build();
    } catch (WriterException e) {
        e.printStackTrace();
    } catch (IOException e) {
        e.printStackTrace();
    } catch (NoSuchAlgorithmException e) {
        e.printStackTrace();
    } catch (InvalidKeySpecException e) {
        e.printStackTrace();
    } catch (Exception e) {
        e.printStackTrace();
    }
    return Response.serverError().build();
}

It is worth noting that the number of catch statements in the try...catch clause will vary and depend on the number and type of exceptions that can be thrown by the underlying methods.

An End-Point Returning a PDF

The last method for today – defining an end-point for returning PDF files – is nearly identical to the one we used for returning images, except for one thing:

• Different MIME type and explicit file encoding in @Produces: @Produces("application/pdf; charset=utf-8").

@GET
@Path("{id}/pdf")
@Produces("application/pdf; charset=utf-8")
public Response createQRPDF(@PathParam("id") String id) {
    try {
        byte[] pdfDocument = pdfDocumentSupplier.pdfDocumentGenerator(id);
        return Response.ok(pdfDocument)
            .header("Content-Disposition", "inline; filename=\"" + id + ".pdf\"")
            .build();
    } catch (WriterException e) {
        e.printStackTrace();
    } catch (IOException e) {
        e.printStackTrace();
    } catch (NoSuchAlgorithmException e) {
        e.printStackTrace();
    } catch (InvalidKeySpecException e) {
        e.printStackTrace();
    } catch (Exception e) {
        e.printStackTrace();
    }
    return Response.serverError().build();
}

Of course, there are also obvious differences in the contents of the byte[] array, but I consider that being outside the scope of this post – you can study those differences on your own.

What’s Next?

Here we have seen how easy it is to define an end-point that can return documents with various MIME types. In the next posts, we will be taking a closer look at things like how you can equip your end-points with metrics, provide auto-generated documentation based on OpenAPI, and add more resilience with fail-over and circuit-breakers.

• Intro
• Choosing a Library
• Generaring PDF Files
  • Adding Maven Dependency
  • Getting Started With the Code
  • Adding Text
  • Adding Images
  • Adding Metadata to the Document
  • Closing, Saving and Returning the Document
• What Now?

Intro

In my previous post, I have started describing a system for “checking-in” to a location using QR codes in Java. Now, I would like to describe another part of that system that will show you how to get started with generating PDF files using Java.

So, let’s have a look at how this can be implemented in your solution – step by step.

Choosing a Library

Generating PDFs is normally something you would like to do using 3rd party libraries, and there are quite a few alternatives available. While choosing a library it might be a good idea to have a closer look at the licensing for that library – some of them might be very permissive and some might force your code to comply to a specific licensing. Some libraries even come with dual licensing, one under a proprietary model and one supporting an open source model.

One should also consider other aspects like maturity and whether it is a high- or low-level library. The latter will tell you how much code you will actually end up writing to implement your features. In the course of this project, I ended up trying two different libraries – iText and, later, Apache PDFBox. Since the point of this code was a tutorial, I decided to stick with PDFBox as it is distributed under more permissive license – Apache License 2.0, as opposed to dual licensed iText that is under AGPL and a commercial license.

Generating PDF Files

As mentioned earlier, this library provides quite extensive functionality for generating PDF files, but it is also quite low-level, so you will have to be prepared to implement a few things you might usually take for granted, e.g. things like calculating coordinates for text that has to be centered on a page and a few other things. However, the library has a great community, so it is quite easy to get help.

Adding Maven Dependency

Ok, let’s get started. First things first, you will need to add the following dependencies to your pom.xml to use PDFBox (assuming you are using Maven to build your project):

    <dependency>
      <groupId>org.apache.pdfbox</groupId>
      <artifactId>pdfbox</artifactId>
      <version>2.0.15</version>
    </dependency>

Getting Started With the Code

For this post, I decided to paste the source code for the whole function doing the PDF generation and separate it with a few sentences, explaining the most interesting parts of the code. You can always piece the code together, or just have a look at the code in my repo. (Bonus: If you are interested in how much work it was to port the code from iText to PDFBox, this commit should give you a rough idea.)

We start with defining a document object, a page object and add a page to a document. Afterwards, we create a content stream object that will be added to the page and document objects. This object will be responsible for holding the text and images we will be generating here.

// Assume that the following variables are declared and set:
//   QRCodeSupplier qrCodeSupplier - to generate QR codes, shown in the previous post
//   String id - a string that will be shown on the top of the PDF file and used in the QR code
//   String timeZone - a string containing current time zone

String headerTitle = id;
PDFont headerFont = PDType1Font.COURIER_BOLD;

int marginTop = 30;
int fontSize = 30;

PDDocument document = new PDDocument();

PDPage page = new PDPage(PDRectangle.A4);
PDRectangle mediaBox = page.getMediaBox();
document.addPage(page);

PDPageContentStream contentStream = new PDPageContentStream(document, 
                                                            page, 
                                                            PDPageContentStream.AppendMode.APPEND, 
                                                            true);

Adding Text

Now we will need to calculate the coordinates for the header text string and make sure it will appear centered independent of the font and size. This is one of the “low-level” parts you will have to deal with when using PDFBox.

// calculate coordinates to center the header text
float titleWidth = headerFont.getStringWidth(headerTitle) / 1000 * fontSize;
float titleHeight = headerFont.getFontDescriptor().getFontBoundingBox().getHeight() / 1000 * fontSize;
float titleStartX = (mediaBox.getWidth() - titleWidth) / 2;
float titleStartY = mediaBox.getHeight() - marginTop - titleHeight;

Next, we will be adding the text itself and setting font and coordinates for it on the page:

// add header text to the document
// Note: This solution will not support fixed-width paragraphs and text flow
contentStream.beginText();
contentStream.setFont(headerFont, fontSize);
contentStream.newLineAtOffset(titleStartX, titleStartY);
contentStream.showText(headerTitle);
contentStream.endText();

Adding Images

Now, let’s examine how to add an image to a PDF document. Here, you can use createFromFile() in case your image is already available, or createFromImage() is you are generating the image on the fly and/or returning it from another function.

Below, you will also find examples of code to scale and to calculate coordinates for centering the image:

// get image as a byte array
ByteArrayInputStream bais = new ByteArrayInputStream(qrCodeSupplier.qrCodeGenerator(id));
BufferedImage bim = ImageIO.read(bais);

// convert image to an object that can be added to the PDF document
PDImageXObject pdImage = LosslessFactory.createFromImage(document, bim);

// calculate coordinates to center the image
float scale = 1f;
int imageOffset = 100;

float imageWidth = pdImage.getWidth() * scale;
float imageHeight = pdImage.getHeight() * scale;
float imageStartX = (mediaBox.getWidth() - imageWidth) / 2;
float imageStartY = titleStartY - imageHeight - imageOffset;

// add image into the document
contentStream.drawImage(pdImage, imageStartX, imageStartY, 
                        pdImage.getWidth() * scale, pdImage.getHeight() * scale);

// closing the stream
contentStream.close();

Adding Metadata to the Document

Here, you can see a few examples of how you can add metadata to your document. This is done with a help of a few methods available thought the API:

// add metadata
document.getDocumentInformation().setTitle("Generated QR code for " + id + ".");
document.getDocumentInformation().setSubject("with a secure string");
document.getDocumentInformation().setAuthor("rm");
document.getDocumentInformation().setCreator("rm");
document.getDocumentInformation().setCreationDate(date);

Closing, Saving and Returning the Document

After your document is built and you have added all the contents, remember to save and close your document. Now you can either save the document to the file with document.save(), or returning the document as a byte array to another function with byteArrayOutputStream.toByteArray():

// save and close document
ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();
document.save(byteArrayOutputStream);
document.close();

// return document as byte[]
return byteArrayOutputStream.toByteArray();

What Now?

In the last two posts, we have seen how to generate QR codes with a hashed string and PDF files with Java. In the next post, I will be showing how to put it all together into a MicroProfile microservice.

• Intro
• Generaring QR Codes
• Hashing Strings
• What Now?

Intro

I have been testing out new functionality for “checking-in” to a location using QR codes. To make sure the user is at the specified location and is scanning my QR code (and not a “fake” code created by someone else), I needed to add a way of “signing” each code with a value that only I – the provider of the QR code – could know. This would also make it simple enough to be able to implement the same mechanism in the app used to scan the codes to verify the validity on the client side.

I ended up with a solution where I would have a QR code containing a JSON object with a Name and a Key – a hashed and salted name string. The string will be read by the client app used to scan the code and hashed using the same algorithm with the same secret salt value, and compared to the value in the QR code on the client side.

The data structure inside a generated QR code would be like this:

{ 
    "Name", "MyString",
    "Key", "HashedMyStringWithSecretSalt"
}

When it comes to the implementation, I decided to do the generation or codes in Java and, later, implement this as a standalone microservice. Here, I must admit that I was surprised by how simple it was using a specialized library. More about that below.

So, let’s have a look at how this can be implemented in your solution – step by step.

Generating QR Codes

First, I needed a library that can handle QR codes, and I decided to use Zebra Crossing (“ZXing”) library because of its simplicity and popularity (i.e. community around it).

All you need to get started is to add the following dependencies to your pom.xml (assuming you are using Maven to build your project):

<dependency>
  <groupId>com.google.zxing</groupId>
  <artifactId>core</artifactId>
  <version>3.4.0</version>
</dependency>
<dependency>
  <groupId>com.google.zxing</groupId>
  <artifactId>javase</artifactId>
  <version>3.4.0</version>
</dependency>

This library provides quite an extensive functionality both for generating and reading codes. This was more than enough for my use case where I just needed to generate a QR code with a simple JSON object:

public byte[] qrCodeGenerator(String id) throws IOException, 
                                                WriterException, 
                                                InvalidKeySpecException, 
                                                NoSuchAlgorithmException {

    String filePath = "QRCode.png";
    String charset = "UTF-8";
    Map hintMap = new HashMap();
    hintMap.put(EncodeHintType.ERROR_CORRECTION, ErrorCorrectionLevel.L);

    Map<String, String> qrCodeDataMap = Map.of(
            "Name", id,
            "Key", keyProvider.generateVerificationKey(id) 
            // see next section for ´generateVerificationKey´ method
    );

    String jsonString = new JSONObject(qrCodeDataMap).toString();
    createQRCode(jsonString, filePath, charset, hintMap, 500, 500);

    BufferedImage image = ImageIO.read(new File(filePath));
    ByteArrayOutputStream baos = new ByteArrayOutputStream();
    ImageIO.write(image, "png", baos);
    byte[] imageData = baos.toByteArray();

    return imageData;
}

private void createQRCode(String qrCodeData, 
                          String filePath, 
                          String charset, 
                          Map hintMap, 
                          int qrCodeHeight, 
                          int qrCodeWidth) throws WriterException, 
                                                  IOException {

    BitMatrix matrix = new MultiFormatWriter().encode(
            new String(qrCodeData.getBytes(charset), charset),
            BarcodeFormat.QR_CODE,
            qrCodeWidth,
            qrCodeHeight,
            hintMap
    );

    MatrixToImageWriter.writeToPath(
            matrix,
            filePath.substring(filePath.lastIndexOf('.') + 1),
            FileSystems.getDefault().getPath(filePath)
    );
}

Note also fun little thing – the conversion of Java hashmaps to a JSON object using JSONObject. Sometimes it is much easier to build up data structure the way you want it, and then serialize to JSON:

Map<String, String> qrCodeDataMap = Map.of(
        "Name", "SampleText",
        "Key", "SomeHashedValue"
);
String jsonString = new JSONObject(qrCodeDataMap).toString();

To be able to use JSONObject class, you would need to add the following dependency to your pom.xml:

<dependency>
  <groupId>org.json</groupId>
  <artifactId>json</artifactId>
  <version>20180813</version>
</dependency>

If you are looking for a more simplified interface, you might also check out QRGen that claims to simplify QR code generation API for Java even further and is built on top ZXing. However, ZXing was absolutely fine in my case.

Hashing Strings

Now, I needed to be able to hash a string in a quick and secure manner. For this, I decided to use the method suggested by OWASP for Java. To implement this method you will need to start with updating your pom.xml:

<dependency>
  <groupId>commons-codec</groupId>
  <artifactId>commons-codec</artifactId>
  <version>1.12</version>
</dependency>

And here is the (somewhat simplified) implmentation of the said method in Java:

public String generateVerificationKey(String str) throws NoSuchAlgorithmException,
                                                         InvalidKeySpecException {
    int iterations = 10000;
    int keyLength = 512;

    char[] strChars = str.toCharArray();
    byte[] saltBytes = salt.getBytes();

    SecretKeyFactory skf = SecretKeyFactory.getInstance("PBKDF2WithHmacSHA512");
    PBEKeySpec spec = new PBEKeySpec(strChars, saltBytes, iterations, keyLength);
    SecretKey key = skf.generateSecret( spec );
    byte[] hashedBytes = key.getEncoded( );

    return Hex.encodeHexString(hashedBytes);
}

What Now?

By now you should be able to generate QR codes with a hashed string. In the next post, I will be sharing code on how to embed and generate PDF files with this information with Java, followed by a post where it all will be put together into a MicroProfile microservice. Stay tuned!

• Intro
• 1. Before you begin
• 2. Create a Cloud Function
• 3. Deploy the Cloud Function

Intro

Recently we decided to migrate our builds from Travis CI to Google Cloud Build to speed up the builds. The process was quite easy and flawless; however, we were still missing a few minor things. One of them was the notifications from Cloud Build to our #ops channel in Slack. This was slightly annoying because you would not know if the build was finished and the site was deployed, or if it failed for some reason.

Integrating with Cloud Build was a bit more different than what you are used to from integrations with Jenkins or Travis CI. Normally you would just create a webhook that would call an interface in the Slack API. In Cloud Build, on the other hand, everything is getting posted to the Pub/Sub queue built into the platform, and here you would just need to subscribe to the specific queue and listen for the events. To achieve the latter, you would need a small serverless function to listen for these events and to call the Slack API.

Note that here we will, technically, be using paid services on Google Cloud Platform, as both Cloud Build, Cloud Pub/Sub, and Cloud Functions are billable components. However, since all the components above provide a generous free tier, you will need to work hard to get passed the free tier with this setup.

• Cloud Build: Free first 120 builds-minutes per day for Basic machine type (n1-standard-1).
• Cloud Pub/Sub: Free first 10GB per month (pricing).
• Cloud Functions: Free first 2 million invocations per month (pricing).

1. Before you begin

1.1 Prepare your GCP project

I assume you have a Google Cloud Account, and that you have signed in to your account.

1. Select or create a Google Cloud Platform project, e.g. from the Manage resources page.
2. Make sure that billing is enabled for your Google Cloud Platform project.
3. Enable the Cloud Functions and Cloud Pub/Sub. You can also enable the APIs using this link.
4. Use Cloud Shell right from the browser, or you can Install and initialize the Cloud SDK on your own machine.
5. If you have installed the Cloud SDK, update and install gcloud components:

gcloud components update &&
gcloud components install alpha beta

1.2 Prepare your Slack App

I assume you have Slack installed and that you have created and signed-in to your account.

Create a new Slack app:

1. Choose the app’s name and your Slack team. Click Create.
2. Click Incoming Webhooks.
3. Activate incoming webhooks.
4. Click Add New Webhook to Workspace. An authorization page opens.
5. From the drop-down menu, select the channel to which you would like notifications sent.
6. Click Authorize.
7. A webhook for your Slack application has been created. Copy the webhook URL and save it for later use.

2. Create a Cloud Function

We need to create a Cloud Storage bucket to stage your Cloud Functions files. Use [STAGING_BUCKET_NAME] that is a globally-unique bucket name (such as [PROJECT-ID]_cloudbuilds):

gsutil mb gs://[STAGING_BUCKET_NAME]

You should see the following output:

Creating gs://[PROJECT-ID]_cloudbuilds/[STAGING_BUCKET_NAME]...

Next, create a directory on your local system for the application code:

mkdir ~/gcb_slack
cd ~/gcb_slack

Then, create the following two files in the gcb_slack directory.

File 1: package.json

{
  "name": "google-container-slack",
  "version": "0.0.1",
  "description": "Slack integration for Google Cloud Build, using Google Cloud Functions",
  "main": "index.js",
  "dependencies": {
    "@slack/client": "4.10.0"
  }
}

File 2: index.js

Note: Make sure to update SLACK_WEBHOOK_URL in the code below.

const IncomingWebhook = require('@slack/client').IncomingWebhook;
const SLACK_WEBHOOK_URL = "<INSERT YOUR WEBHOOK FROM STEP 1.2>"

const webhook = new IncomingWebhook(SLACK_WEBHOOK_URL);

// subscribe is the main function called by Cloud Functions.
module.exports.subscribe = (event, callback) => {
 const build = eventToBuild(event.data.data);

  // Skip if the current status is not in the status list.
  // Add additional statues to list if you'd like:
  // QUEUED, WORKING, SUCCESS, FAILURE,
  // INTERNAL_ERROR, TIMEOUT, CANCELLED
  const status = ['SUCCESS', 'FAILURE', 'INTERNAL_ERROR', 'TIMEOUT'];
  if (status.indexOf(build.status) === -1) {
    return callback();
  }

  // Send message to Slack.
  const message = createSlackMessage(build);
  webhook.send(message, callback);
};

// eventToBuild transforms pubsub event message to a build object.
const eventToBuild = (data) => {
  return JSON.parse(new Buffer(data, 'base64').toString());
}

// createSlackMessage create a message from a build object.
const createSlackMessage = (build) => {
  let message = {
   text: `Build \`${build.id}\``,
    mrkdwn: true,
    attachments: [
      {
        title: 'Build logs - Your Custom Message Goes Here',
        title_link: build.logUrl,
        fields: [{
          title: 'Status',
          value: build.status
        }]
      }
    ]
  };
  return message
}

3. Deploy the Cloud Function

To deploy the subscribe function with a Cloud Pub/Sub trigger, run the following command in the gcb_slack directory:

gcloud functions deploy subscribe --stage-bucket [STAGING_BUCKET_NAME] \
    --trigger-topic cloud-builds

where [STAGING_BUCKET_NAME] is the name of your staging Cloud Storage Bucket that you defined earlier.

You should see an output confirming the creation of the cloud function and status: READY.

After you’ve completed deployment of the Cloud Function, when a build event occurs, you will receive a Slack notification.

Also, feel free to customize your app, like adding a custom icon, as I did with mine. ☝️

• The Beginning
• The New Level
• Story 1: An Icon of Norwegian IT History
• Story 2: A Machine for 4.300 USD
• Story 3: Two “Portable” Computers
• Story 4: A Very Special Machine
• Stranger Strings: The Movie

You may already have seen Stranger Strings, a tribute to the Stranger Things series, that was created for JavaZone 2018. This was our latest addition to the long list of short films we have created throughout the years.

Here, I would like to share the untold story of the computers and hardware from the movie that might have been seen merely as the props in the movie to the untrained eye, but when you look a bit deeper have quite interesting stories to tell. JavaZone has a tradition of creating short videos before the conference and most of the videos – with a very few exceptions – are in the format of trailers or short movies. These are usually geeky parodies to well-known (and usually well-loved) movies or series. This time we decided to do something a bit different. We created a slightly longer film showing us a parallel, and geekier, Stranger Things universe, and called it Stranger Strings.

The Beginning

Easter Eggs: Gotta Collect ‘em All!

We had an amazing JavaZone team, film crew, and actors. It was great fun being a part of the team creating the video and we worked hard on trying to add as many fun references and puns as possible without going over the top. We also did include a hint to a mini online adventure game made by our team and a puzzle that would lead you to a surprise (I still won’t disclose that one!). In case you haven’t seen it yet, you will find a link to the game and hints by just watching the movie a little bit closer.

Some of these things were right out in the open, in plain sight, while others were much subtler. Nonetheless, with 1.000 views a day on the first week, and a bit short of 20.000 views in total as of today, most of the Easter eggs and references were noticed by quite a few people.

The New Level

However, the movie still has something else that was a bit more difficult to notice simply by watching it, and I think it is finally about time to reveal those things as well.

When setting up the set for this movie one of the things we had to do was to find computers from the correct time period, and we needed quite a few of them. So, after asking around on social media, we were able to get hold of a few machines. And what machines they were!

This is where it all started, and where our movie for 2018 got yet another layer of hidden features.

We got all the machines from the collection of the University of Oslo Library, Department of Informatics (again, thanks a lot for providing those!). And all these machines had quite an interesting story to tell.

So, in addition to serving the main purpose of being, well, the machines from the time period for the movie set, they also brought a tiny bit of the Norwegian computer history to the movie.

(Spoiler alert: One could argue that at least one of the machines had a role in the development of Simula. Excited to know which, or what even Simula is? Read on!)

Now, let’s have a look at some of the highlights of what we had at the set.

Story 1: An Icon of Norwegian IT History

Control Panel for NORD-10/S (Norsk Data)

The first one out is the control panel from Norsk Data’s NORD-10/S. We used it as a part of the “mainframe” in the movie. It might not have been very visible in the movie, but it still had a great symbolic value.

It was great to be able to include a piece of that history into the Stranger Strings. What makes this piece special is that the company had quite a significant role in the Norwegian IT history.

There are quite a few interesting facts connected to Norsk Data. It was a significant supplier of minicomputers to many research projects, in particular to CERN in Geneva, Switzerland. It was also the first Norwegian company to get listed on the London Stock Exchange (1981) and NASDAQ (1983) – just to mention a few.

Also, this particular piece of hardware was acquired by the University of Oslo in 1979 for the university’s main library located off the new, main campus, at Drammensveien (Norwegian article). It came with all the cool stuff that might want back then, like the state-of-the-art punch card reader, and it was used to connect the library with a direct line to the rest of the university network.

Story 2: A Machine for 4.300 USD

Atari Mega ST2

Now, let’s talk about Will’s machine – an Atari Mega ST2. It was packing an 8 MHz Motorola 68000 processor, had a whopping 2 MB RAM, and 20 MB hard drive.

This machine was one of the 35 machines bought by the Department of Informatics at the University of Oslo in May 1988 and came with an impressive price tag of 18.360 NOK a piece (which corresponds to 36.269 NOK, or 4.297 USD in today’s equivalent).

So, we can surely say that this machine did play a role in the education of a few generations of IT professionals here in Norway, before starring in a movie exactly 30 years later.

Story 3: Two “Portable” Computers

IBM Portable Personal Computer

We also got to borrow not one, but two of IBM’s portable personal computers. This model, form 1984, was the IBM’s first portable machine, had an 8088 processor, and weighed approximately 15 kg (33 lbs).

Yes, portable meant something else back in the days…

Story 4: A Very Special Machine

IBM XT

I saved the best for last! The machine you could often see in the movie placed between two portable IBMs, and the main machine of one of the main characters – Three – is an IBM XT.

IBM XT was one of IBM’s first PC models. It came with two floppy drives for 5.25” disks and 256 KB RAM. This particular machine was later upgraded to 640 KB RAM and one of the floppy drives was replaced with a 20 MB hard drive.

So, what is so important about this machine, you might ask?

The most interesting thing about this particular machine is that it belonged to Kristen Nygaard – the co-inventor of object-oriented programming and the programming language Simula, together with Ole-Johan Dahl.

Simula was the first object-oriented programming language and the language that is considered to have had a strong influence on Java and many other modern object-oriented languages.

So, one can argue that this machine, being the office computer of Kristen Nygaard, had a direct role in the development of Simula and work related to it.

Finally, I would like to yet again thank everyone involved in the creation of this movie! You are all awesome and this would not have been as fun and great without you!

Stranger Strings: The Movie

Now, that you have read the stories behind some of the machines in the movie, it is time to watch it again.

Containers are something that we use to run our applications and, normally, we dispose of the whole container when we build a new version of the application or need to upgrade something in the setup. This means that containers are generally having a short lifespan.

However, in this case, I want to show you how to build something that exists for an even shorter period of time and that can be used as an alternative to a local setup for building and testing applications locally before pushing it to test, staging, production, etc.

This is a simplified example of what is being done on a much bigger scale with moving your CI/CD pipelines to such disposable containers, and with libraries like Testcontainers.

In this case, I would like to show you how to setup Jekyll applications, but this can be easily applied to any kind of applications written in any of your favorite languages, like Java or Python. Until recently, I have been running a Jekyll installation locally with all dependencies installed on my machine. However, it has been a bit challenging when moving between machines and reinstalling operating systems. To simplify the process, I decided to containerize the local build and test processes.

I wanted the following:

• To build my code from and to the local folder on my (host) machine
• Run the application (in this case this blog) from a local folder on my (host) machine
• Avoid setting up the environment, or have a minimal and portable setup
• Avoid environment clean-up – I didn’t want to hold on to the unnecessary containers and container images

TL;DR: The solution

(see next section for the explanation)

$ export JEKYLL_VERSION=3.8
$ docker run --rm --volume="$PWD:/srv/jekyll" \
       -it jekyll/jekyll:$JEKYLL_VERSION jekyll build
$ docker run --name newblog --volume="$PWD:/srv/jekyll" -p 4000:4000 \
       -it jekyll/jekyll:$JEKYLL_VERSION jekyll serve --watch --drafts

Explanation – line by line

So, let’s take a closer look at each of the lines:

1: export JEKYLL_VERSION=3.8

Just setting up versions that will be used later – a bit of housekeeping. Nothing exciting here.

2: docker run --rm --volume="$PWD:/srv/jekyll" \
        -it jekyll/jekyll:$JEKYLL_VERSION jekyll build

Here, we build the code and output it to the same disk volume as the source code, i.e. the volume that is shared with my host machine. Now I have the built version on my machine without the hassle of setting up the local build environment. In addition to that, I will be doing some clean-up, by deleting the build container after the build job is finished.

• --rm – just execute the command and clean-up (remove the container, file system, etc.)
• --volume – mapping the current directory to /srv/jekyll in the container
• -it instructs Docker to allocate a pseudo-TTY connected to the container’s stdin; creating an interactive shell in the container
  • -i – attach container’s STDIN
  • -t – allocate a pseudo-TTY
• jekyll/jekyll:$JEKYLL_VERSION – Docker image to use and the tag
• jekyll build – command to run

3. docker run --name newblog --volume="$PWD:/srv/jekyll" -p 4000:4000 \
        -it jekyll/jekyll:$JEKYLL_VERSION jekyll serve --watch --drafts

This will create another container that will be running our application. Here we will need to add a few other parameters – like mapping the container ports to the ports on the local machine and giving the container a name.

• --name newblog – give your container a name
• --volume – mapping the current directory to /srv/jekyll in the container
• -p – bind port 4000 of the container to TCP port 4000 (-p host_machine:container)
• -it instructs Docker to allocate a pseudo-TTY connected to the container’s stdin; creating an interactive shell in the container
  • -i – attach container’s STDIN
  • -t – allocate a pseudo-TTY
• jekyll/jekyll:$JEKYLL_VERSION – Docker image to use and the tag
• jekyll serve --watch --drafts – command to run

Now you can stop the container with CTRL+c, and restart it again with:

$ docker start newblog -i

If you don’t want the container being persistent on your system, you can simply add --rm as in the previous command:

$ docker run --rm --name newblog --volume="$PWD:/srv/jekyll" -p 4000:4000 \
       -it jekyll/jekyll:$JEKYLL_VERSION jekyll serve --watch --drafts

• Moving Around the Command Line
• Editing Commands in the Command Line
• Bonus

Using the command line can simplify and even automate many of the operations we do on a computer. However, using the command line can mean quite a bit of typing and a possibly large number of parameters. In this post, I would like to focus on how to navigate the cursor and edit the command line, while leaving all the other Bash tricks for the future posts.

I also have created simple graphics to illustrate some of the main shortcuts listed below. This (hi-res) image can be printed for future reference.

Note: Please note that all commands containing ALT combinations might not work depending on your system configuration, and most definitely not work on MacOS. Normally, it is because these combinations are mapped to something else. However, you can still use the same shortcuts simply by replacing ALT with ESC.

Moving Around the Command Line

So, let’s first speak about how to move the cursor around – because using just arrow keys is often not the most optimal way of navigating. Sometimes you might want to go to the beginning of the line, to the end of the line, or simply jump from one word to another, where word – in this context – is set of characters separated by spaces (or sometimes other special characters), or as documentation states it:

A sequence of characters considered as a single unit by the shell. Also known as a token.

# Moving the cursor – fast
CTRL+a         Go to the beginning of the line (same as Home)
CTRL+e         Go to the End of the line (same as End)
ALT+b / ESC+b  Go one word back (to the left)
ALT+f / ESC+f  Go one word forward (to the right)

# Moving the cursor – one character at a time
CTRL+f         Go forward one character
CTRL+b         Go backward one character

# Using history
CTRL+r         Backwards search in previously executed commands (history)
CTRL+p         Previous command (same as Up arrow)
CTRL+n         Next command (same as Down arrow)

Editing Commands in the Command Line

Now that we are able to navigate freely along the command line, it is time to do some modifications. Here, we will see how to delete, cut, paste, and swap words and characters.

# Deleting whole words
ALT+Del        Delete the word before (to the left of) the cursor
ALT+d / ESC+d  Delete the word after (to the right of) the cursor
CTRL+w         Cut the word before the cursor to the clipboard

# Deleting parts of the line
CTRL+k         Cut the line after the cursor to the clipboard
CTRL+u         Cut/delete the line before the cursor to the clipboard

# Deleting single characters
CTRL+d         Delete character under the cursor (same as Delete key)
CTRL+h         Delete character before the cursor (same as Backspace key)

# Paste, Undo, revert, and more
CTRL+l         Clear the screen (similar to the 'clear' command)
CTRL+y         Paste the last thing to be cut (yank)
CTRL+_         Undo
ALT+r / ESC+r  Revert the changes and replace with the line as it was 
                in History.

# Swap 'em!
CTRL+t         Swap the last two characters before the cursor
ALT+t / ESC+t  Swap current word with previous
 
# Convert to UPPER, lower, or Sentence case
ALT+u / ESC+u  Capitalise characters from the cursor to the end of 
                the current word and move to the end of the word.
ALT+l / ESC+l  Lower the case of characters from the cursor to the
                end of the current word and move to the end of the word.
ALT+c / ESC+c  Capitalize the character under the cursor position 
                and move to the end of the word.

Bonus

First, the most obvious – you can always find more gems in the man pages for Bash both in your terminal and online (for instance on this mirror). To view it in your terminal, type:

$ man bash

Now, over to something different. Since we have been talking about the command line and shells it is worth mentioning some less-known (and sometimes “as a curiosity”) shortcuts in another terminal – Command Prompt, cmd.exe:

Function keys in cmd.exe:
  - F1: Pastes the last executed command (character by character)
  - F2: Pastes the last executed command (up to the entered character)
  - F3: Pastes the last executed command
  - F4: Deletes current prompt text up to the entered character
  - F5: Pastes recently executed commands (does not cycle)
  - F6: Pastes ^Z to the prompt
  - F7: Displays a selectable list of previously executed commands
  - F8: Pastes recently executed commands (cycles)
  - F9: Asks for the number of the command from the F7 list to paste

Good luck! Try them out and let me know how that goes!

• The Existing State of Affairs
• The Moving Parts
• Conclusion

Let’s take a look into what we can do to achieve a better development environment than an average development project – a project that most of us have seen at some point in our professional lives, or maybe even are a part of right now. We will also look into some tools and patterns that will help us convert those projects into a paradise for the developers.

Just a few decades ago, we were working in ways that might look like unproductive, in the best case. Our development models were predominated by waterfalls, our IDEs were basic and we were compiling our projects by hand, using javac, or building up the CLASSPATH depending on the GOTO statements in a huge spaghetti code contained in countless bat files. Our code lived in a very simple versioning systems that were not distributed or supported branching strategies that are praised by the developers today. Our documentation lived in doc files on shared network drives, side by side with the simple issue tracking systems, that don’t even get close to what we have today.

Today, it is all different – we have Git, real issue tracking, IDEs, all that integrated with build servers and collaborative platforms. Yes, everything is much better, more effective and user-friendly, one might think that we are in the paradise already? Well… yes, things are fortunately getting better, however, we are still doing things in a way that might still give you nightmares, several decades from now.

Last years I have been working and invited to evaluate and help with an audit of various projects. Here are some of my observations and thoughts.

The Existing State of Affairs

This tweet describes it pretty well:

YOU ARE IN A LEGACY CODEBASE
> RUN TESTS
YOU HAVE NO TESTS
> READ SPEC
YOU HAVE NO SPEC
> WRITE FIX
YOU ARE EATEN BY AN ELDER CODE HACK.

Some of the issues are, naturally, remnants of the past – the legacy systems; but even those systems and most of the other problems we see today can be avoided if we slightly change our view at some of the main parts of the development process. In most of the cases, we would be aware of those issues, but we might need to explain and motivate the others – often people are responsible for the projects and those who prioritize the development and maintenance backlog.

The Moving Parts

The road to a great nightmare-free future consists of three components: the code quality, the development and build tools, and a good documentation and collaboration systems. When evaluating systems I often start asking some simple questions listed below to get an idea of the system.

1. The Code

The Code Quality

First thing off is the general code quality. I often start by asking about simple things – if the project has a coding standard, and if it is being followed. I also ask to take a quick peek at the code and check minor things like file encodings and MIME types. I also follow up with a question if the team is practicing code reviews, and how they are doing that.

While those things alone don’t have to mean anything, and are minor issues individually, together with other factors they still are initial indicators of possible neglect. This gives me a possibility to map areas where to look further.

In addition to that, there are also some more specific parts that will be listed as sub-sections below.

The third-party libraries

The role of the third party libraries and their use is often forgotten and neglected when considering code and system quality. This is quite unfortunate as this is the part of the code that you might not be able to patch easily, and is harder to maintain compared to your own codebase. Here are some simple questions that might help with getting a better grip on third-party libraries:

• Do you keep track of your third-party libraries?
• Do you regularly check if there are known issues or vulnerabilities in them?
• Do you have a plan for keeping them updated?
• Are the libraries you are using being actively maintained by the authors?
• Are the libraries you are using compatible with each other?
• Do the libraries you are using have appropriate licenses that are compatible with your system? (This also applies to the open source software licenses.)

Issues and vulnerabilities are being found and patched all the time. As an example for this, let me point to Google’s OSS-Fuzz Project that has found numerous security vulnerabilities in several critical open source projects. Unfortunately, even though many people are aware of the security issues in software in general, the library updates still often tend to be forgotten.

It is also worth noting that while most of the issues on my list above are security related, the last one might be of a legal sort, and probably is the most neglected of the issues listed.

The Architecture

I am often being asked to assess a system and tell something about its architecture compared to more modern systems. The different aspects of the system’s architecture will tell a lot about its maintainability both when it comes to further development, bug fixing, and keeping the system running. Some questions that might help with determining the state of the system would be:

• Does your architecture support automated deployment?
• Does your architecture support continuous deploy and delivery?
• Does your architecture support load balancing?
• Does your architecture support microservices?
• How is the architecture implemented in the code?

Tools for Maintaining the Code Quality

Some of this might sound familiar to you when you think of one or several systems you had been working with and you might now be wondering what you can do to improve the code quality? Further steps here would be starting to use proper tools that will be able to tell more about the various aspects of your code. Some of them can be used as plugins to your build system (like Maven), and some could be stand-alone tools.

Stand-alone tools for code analysis:

• SonarQube
• PMD
• FindBugs

Maven plugins to consider (more about plugins in my previous post):

• Assembly
• Versions
• Dependency
• Enforcer
• Surefire
• Failsafe
• Sonar
• Findbugs
• pmd

Bonus: See this post on command line tools for Java projects.

2. Development Tools and Strategies

Now, let’s talk about the development tools. All that fancy code and great architecture will not bring you any closer to a developer’s paradise if there will not be some proper tools to support the development. The code should live in a proper version control system that supports collaboration and things like branching and tagging. There should also be tools that help you with code quality analysis, static code analysis, etc. A good starting point here would be to start with answering the following about the project in question:

• Do you use a proper code versioning tool – Git, or even SVN?
• Do you have a branching (and tagging) strategy?
• Do you have a way of measuring code complexity?
• Do you have a way of measuring test coverage and results?
• Do you run static code analysis?

Some tools that can help you here (again, for Java-based systems):

• IDEs and IDE plug-ins that can do checks at commits, integrate with test and QA tools, etc.
• Build tools: Maven, Gradle, etc.
• Continuous integration tools: Jenkins, TeamCity, Bamboo, etc.
• Frameworks and tools for testing: to run unit tests, integration tests, UI tests, and end-to-end tests.

There are some further strategies and questions to consider:

• Are your environments easy to reproduce with minimal efforts – can you rebuild it by simply running a script?
• Do you have a proper pipeline from packaging, to delivery, to deploy?
• To what environments can you deploy automatically? With the same script, or command?
• Are your environments (like development, testing, staging, pre-prod, production) similar to each other?
• Do you follow the same process to deploy to each environment?
• Are QA and production running on physically separate hardware?
• Are you monitoring all of the environments? (i.e. are you able to see errors before they make it to QA or even production?)

3. Documentation and Collaboration Tools

Last, but not least, we will need to talk about the tools for collaboration and documentation. Without these tools, we will be back to the way things were several decades ago – with documents on shared network drives and other horrors of the 90’s that I mentioned at the beginning of this post. However, good wikis, other collaboration tools, and proper issue tracking will bring your software to another level, encouraging continuous improvement of the system.

• Wiki
• Collaboration – chat, etc.
• Issue tracking tools

No matter how obvious it might seem, it is still important to note that one should avoid multiple documentation and issue tracking systems. Unfortunately, even though it might sound obvious, it is more common than you think – I have seen my share of systems for documentation and issue tracking resulting in fragmented information and confusion.

Conclusion

There are several challenges connected with having and maintaining the good code quality. The first challenge is that a good code quality is not something you can achieve overnight. It takes time and energy to achieve that and it is a continuous process. You will need some tools, techniques, and methodology to prevail, and it will probably be easier to introduce all that from the beginning of a project.

The second challenge would be that it might be hard to convince the stakeholders of the project to invest time and resources into something that does not bring any visible improvements to the table – things like new features and bug fixes are more likely to get prioritized over something that cannot be easily measured.

Actually, while presenting on this topic at JavaOne 2017 in San Francisco, several of the attendees asked me about the ways of getting to a beautiful nightmare-free code and infrastructure, and the ways of convincing the stakeholders that this is the way to go. Unfortunately, there is no one simple solution to this, and the most valuable thing, in this case, would be to show the real value of the good quality code.

The measurements parameters to show the value can be:

• time it takes from the code is written to deploy,
• system stability,
• how often bugs are reported compared to earlier, or
• frequency of errors in logs.

So, what can you do as a developer on a project that might need some help, you might ask? You can just start by continuously suggesting improvements and showing their value to the customer, or the project manager. Now you just need to keep going and gradually improving the system, one small bit at a time.

• Getting Started
• Commands, Files, and Folders Inside a Container
• Cleanup

In this post, I decided to share some of the basic commands you might need to get started with Docker. This is neither an extensive list of the commands available, nor all of the commands you might need. This is merely me sharing a prettified list of my cheat sheet for Docker basics with you.

Getting Started

Before we get started, it might be a good idea to note that all of the commands below are written without sudo. If your installation is not running without sudo (assuming that you are running Linux), you might want to check out the post-installation guide for Linux in the Docker docs.

1. Check if Everything Works

First things first, you can use this simple command to check that your installation is fine. Note: Make sure you have right CPU architecture for your images. Raspberry Pi (ARM) things will not run on x86 architecture, and vice versa.

For x86:

$ docker run docker/whalesay cowsay Hello World!

For Raspberry Pi / AMD:

$ docker run -d -p 80:80 hypriot/rpi-busybox-httpd

2. List Containers

After creating containers, first thing you might want to do is to see what containers you have up and running. To list all running containers you can use:

$ docker ps

This command will give you a list similar to this:

CONTAINER ID        IMAGE                       COMMAND                  CREATED             STATUS                    PORTS                NAMES
e85753d57a67        easypi/dokuwiki-arm         "/bin/sh -c 'php-f..."   1 days ago          Up 23 hours               0.0.0.0:80->80/tcp   mywiki

However, it will not show you any stopped containers. To list all local containers use the -a option:

$ docker ps -a

The output will be more like this (note that is shows also stopped, or even failed containers):

CONTAINER ID        IMAGE                       COMMAND                  CREATED             STATUS                    PORTS                NAMES
573193cf1d5e        hypriot/rpi-busybox-httpd   "/bin/busybox http..."   2 days ago          Exited (0) 5 hours ago                         mytest
e85753d57a67        easypi/dokuwiki-arm         "/bin/sh -c 'php-f..."   1 days ago          Up 23 hours               0.0.0.0:80->80/tcp   mywiki

More on docker ps in the Docker docs.

3. List Images

To list all the images available on your system, simply do this:

$ docker images

4. Containers vs. Images?

What is the difference between containers and images, you might wonder? Well, I have a link for you. This will hopefully help you to understand how Docker manages the data within your images and containers.

5. Starting and Stopping Containers

Another two basic commands – starting and stopping containers:

$ docker start <container_id>
$ docker stop <container_id>

Note: The docker run command first creates a writeable container layer over the specified image, and then starts it using the specified command. That is, docker run is equivalent to the API’s /containers/create, and then /containers/<id>/start.

Commands, Files, and Folders Inside a Container

6. Run Any Command from a Container

You can run any command in a running container just knowing its ID (or name):

$ docker exec -it <container_id_or_name> echo "Hello from container!"

7. Getting Into Containers

Since you can run any command, then you can (obviously) also run a shell from a container; if you have any. This will be a bit similar to running an ssh command to connect remotely to a regular Linux box (given you have bash or sh in the container):

$ docker exec -it <container_id_or_name> bash
$ # or:
$ docker exec -it <container_id_or_name> sh

8. Copy Files From and To Containers

Another useful trick you might need is to copy some files to and from a container. Your friend here is the docker cp command (link to the docs):

$ # To container:
$ docker cp foo.txt <container_name>:/foo.txt
$ # From container:
$ docker cp <container_name>:/foo.txt foo.txt

Cleanup

After playing round with all the images and containers, you might realize that you have quite a collection of these on your drive, just taking up space.

9. Remove Containers

To remove the unused or unwanted containers, you can run the docker rm command with the IDs of those images. The IDs can be retrieved with the docker ps -a command, mentioned above.

$ docker rm <container_id>

10. Remove Images

The docker rmi command followed by the IDs of images will help you to remove the unused or unwanted images. The abovementioned docker images command will help you finding the correct IDs for the images in question.

$ docker rmi <container_id>

Have fun!

• The Story
• Information Security
• Norwegian Personal Data Act (Personopplysningsloven)
• EU GDPR
• The Bottom Line

The Story

It all started when I had to deliver my Apple device for service due to some hardware issues to one of the biggest Apple Premium Resellers and service centers in Norway. After handing in the product, I got an SMS and an email containing a link to a website where I could track the progress of the service online. So far, so good.

While logging in I realized that I already had an account, but did not have the password, so I decided to reset that – that’s where it all started.

The first thing I did, was to push the big, blue “Forgot My Password” button. Unsure if I had to type in the email first, or if I would be forwarded to another page, where I would have to provide my account details to process with the password reset procedure, I just clicked the button.

However, instead of being redirected to a new page, or getting an error about the missing e-mail in the form, I was presented with this page. Are you noticing anything strange?

Well, yes, the site reset the password and sent it over to an email and as an SMS. Cool! The only problem was that at that point it could not have any idea who I was, since I have not provided any information about myself yet, and there were no cookies to identify myself to that site.

Another problem there was that the phone number is shown in clear text (hidden here) was not mine. So, I just reset the password and sent it over to some random user – possibly the first, or the last one in the users table. I tried a few times just to make sure that it was not my fault, and I was still resetting the password for the same person (sorry, total stranger!).

Having worked with systems development for quite some time, I shrug my shoulders, slightly shook my head, mumbled something about weird bugs and reset my password. This time by providing my e-mail address, proceeding to check the status of my device.

Then, it suddenly hit me. By only providing an email to a service, I could see a confirmation about my password is sent to my mobile phone number registered in the system – in clear text!

While the first bug (resetting the password for a random person) might be just annoying to a small group of users, the second one (dumping the phone numbers from the database in clear text) was much worse for a bigger group of people. Why might you ask?

Information Security

Well, given the fact that the company is being one of the biggest service centers for Apple products, it is very likely to assume that many people would have owned, and sent in for a service an Apple device at some point in the past; thereby getting registered in the service provider’s database.

So, now I was sitting in front of an unintentional yellow pages (a.k.a. phone directory) service that could provide me with phone numbers of nearly anyone I wanted by just manually typing their emails, or by creating a script that would try scrape the Internet, or just simply construct emails by putting together firstname.lastname and some @provider.com, and dumping all the phone numbers from their customer database.

Well, of course, bugs happen, so I don’t want to jump into conclusions about the lack of proper testing or similar in general.

However, when we provide data to a company, we expect them to handle it with integrity and care, and not leak personal data to the outside world. While phone number might be considered a low-risk data to be leaked for most of us, it might still be quite sensitive for some groups of people, like some high-profile politicians, celebrities, or anybody else how might have a wish, or even a need, to hide their contact information.

Norwegian Personal Data Act (Personopplysningsloven)

Also, according to The Norwegian Data Protection Authority (Datatilsynet), any information that can be used to identify a person is considered personal. Further, Personal Data Act chapter 2, section 13 (Norwegian: [Personopplysningsloven] from 2000) requires that “the processor shall by means of planned, systematic measures ensure satisfactory data security with regard to confidentiality, integrity, and accessibility in connection with the processing of personal data”.

Further, according to the Personal Data Act section 46, The Norwegian Data Protection Authority (Datatilsynet) may impose a fee for violations of the act, or the regulations, with an amount up to ten times the basic amount of the National Insurance, equivalent to 925,760 NOK (as of May 2017).

EU GDPR

If the fines mentioned above sound bad, just wait to see how expensive it will get with the introduction of EU General Data Protection Regulation (GDPR) next year.

With the introduction of GDPR in 2018, the maximum amount of fines will be raised significantly with an upper limit of 20 million NOK, or the company’s 4% of the total global annual turnover in the previous fiscal year, if this is higher (GDPR art. 83, item 5).

The Bottom Line

Storing any personal information is an important task and requires rigorous testing and planning on what data you collect, why and how it is protected. Deviating from that can be rather harmful to your company both in regard to reputation and the financial penalties.

That all being said, it is important to note that all of the problems I reported to the company in question were fixed within a few hours. However, I don’t know for how long that data was available online, and if anyone had taken advantage of the vulnerability of the system.

Least but not last, I would like to thank the company, and especially the company’s CIO for great communication and quick responses and fast bug fixes.

• Introduction
• Create the Service File
• Activate the Service

Introduction

After your Docker containers are set up and running, you might need to be able to start some of them automatically on a reboot or a crash. There are several ways of getting this done.

One of them is to use restart policies provided by Docker. They can be set to control whether your containers start automatically when they exit, or when Docker restarts.

Alternatively, you can use a process manager such as upstart, systemd, or supervisor instead. In this post, I want to show you how it is done with systemd.

Create the Service File

To create a service file that will be used by systemd (systemctl command), we will first need to get your container name. This can be done by running the following command in your shell:

$ docker ps -a

The output will look something like this. Select the right container from the list, and note its name in the last column. In this example, we will be using mywiki container.

CONTAINER ID        IMAGE                       COMMAND                  CREATED             STATUS                    PORTS                NAMES
573193cf1d5e        hypriot/rpi-busybox-httpd   "/bin/busybox http..."   2 days ago          Exited (0) 5 hours ago                         mytest
e85753d57a67        easypi/dokuwiki-arm         "/bin/sh -c 'php-f..."   1 days ago          Up 23 hours               0.0.0.0:80->80/tcp   mywiki

Now, we will need to create a file (choose an appropriate file name for the service):

$ sudo nano /etc/systemd/system/docker-dokuwiki.service

Paste the following into the file. Set a proper Description, and make sure to update the container name in ExecStart and ExecStop:

[Unit]
Description=DokuWiki Container
Requires=docker.service
After=docker.service

[Service]
Restart=always
ExecStart=/usr/bin/docker start -a mywiki
ExecStop=/usr/bin/docker stop -t 2 mywiki

[Install]
WantedBy=local.target

A couple of notes about the script above:

1. This file is called a unit file for systemd.
2. Make sure you don’t have any extra line brakes within the sections, like Unit, or Service.
3. The -a option in the Docker command for ExecStart makes sure it is running in attached mode, i.e., attaching STDOUT/STDERR and forwarding signals.
4. The -t option in the Docker command for ExecStop specifies seconds to wait for it to stop before killing the container.

Activate the Service

Before we can activate the service we have created, we need to reload the unit file. You will also need to run this command anytime you do any modifications to the unit files:

$ sudo systemctl daemon-reload

To activate the service run the following commands (remember to change the service name):

$ sudo systemctl start docker-dokuwiki.service
$ sudo systemctl enable docker-dokuwiki.service

To disable the service run the following commands (remember to change the service name):

$ sudo systemctl stop docker-dokuwiki.service
$ sudo systemctl disable docker-dokuwiki.service

Changes will come to effect on a reboot:

$ sudo reboot

Now you should have a container that will start on a server reboot, Docker restart, or a crash. Congratulations!

As a next step, you might want to look at (external documentation links):

• Adding some more parameters to the unit file.
• Available docker start options.
• Available docker start options.

• Introduction
• Directory Structure
• Code Metrics
• Encoding and MIME types
• Dependencies
• SonarQube

Introduction

This post will give you an overview of some command line tools that will be able to help you to get the feeling on how your project is doing. Most of the tools are widely available in the main Linux distributions and MacOS (some of them might need installation of Homebrew for Mac). On Windows it can be also made available via Cygwin, or Bash on Windows.

The commands below have been run and tested on a MacOS machine, and might sometimes need some slight modifications to be run on a Linux or a Windows machine. However, it should be able to give an idea of what it is possible to do with these tools. I will also be linking to the documentation for each of the programs in the post.

Directory Structure

Let’s start with something simple. Sometimes all you want to see is the contents and structure of the project without leaving the command line. The tree command might be able to help you out here.

It is highly customizable, takes lots of parameters, and is widely available for an installation via most of the package mangers. Command Prompt on Windows also contains a native alternative with the same name.

$ tree sourcecodefolder/ -L 1 -d
sourcecodefolder
├── src
├── tests
├── sql
└── docs

Code Metrics

If you want to see some metrics about the code in your project, like what languages that are used in the project, as well as information about number files, blank lines, comments, and lines of code, you might want to try the cloc command (it stands for Count Lines of Code).

Simply install the program and point it at the project directory, or a zip file:

$ cloc sourcecodefolder/

It will give you a result that might look something like this:

-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Java                            33           1226           1026           3017
Python                           4            327            337            888
Markdown                         1             11              0             28
SQL                             10             10             12            212
-------------------------------------------------------------------------------
SUM:                            48           1574           1375           4145
-------------------------------------------------------------------------------

Also, it is possible to show all the information by percent:

$ cloc --by-percent cmb sourcecode.zip

Encoding and MIME types

Files, within a single project, with different, or wrong, encoding might mean trouble with showing non-ASCII characters correctly. They might even give you compilation errors. Therefore, finding and marking those files as quickly as possible might help you manage your project.

While MIME types, on the other hand, are not as bad at causing trouble, the diversity of the MIME types among the same kind of files, like *.java files in this example, might indicate a lack of a standard for developer tools and code standard in the project.

This kind of information can be extracted with file command for each file, or it might be done a bit more automatic for the whole project, and it might look like this:

$ find . -name *.java | \
    xargs file -I $1 | \ 
    gawk 'match($0, /.* (.*); charset=(.*)$/, gr) {print gr[2]}' | \
    sort | uniq -c

A short explanation for the script:

1. find all java files recursively (find command)
2. process them with the file command
3. grab the output and extract data with regex (gawk)
4. sort, extract unique values and count (sort, uniq)

The result will look something like this:

 257 iso-8859-1
 117 unknown-8bit
 678 us-ascii

If you want to drill down a bit further and search for both MIME types and encoding, showing the distribution of encoding per MIME type, you can also extract that from the information supplied by the file command like this:

$ find . -name *.java | \ 
    xargs file -I $1 | \
    gawk 'match($0, /.* (.*); charset=(.*)$/, gr) \
    {print gr[1] " --- " gr[2]}' | \
    sort | uniq -c

A short explanation for the script is almost the same as above. The only difference is that we use information from both of the regex groups (gr[1], and gr[2]). The result will look something like this:

  15 text/html --- iso-8859-1
  34 text/html --- us-ascii
 134 text/plain --- iso-8859-1
  16 text/plain --- unknown-8bit
   1 text/plain --- us-ascii
   1 text/x-c --- iso-8859-1
   8 text/x-c --- us-ascii
   5 text/x-c++ --- iso-8859-1
   6 text/x-c++ --- us-ascii

Dependencies

Now, over to a bit more advanced stuff – analyzing dependencies in your project. Why would you want to do that? Well, in short: dependencies increase complexity, and cyclic dependencies are bad for your project and your health. They also hurt your modularity and complicate the build process.

This kind of analysis can be done with jdepend. It is easy to run and can be run both separately, and as a part of a build tool, like Maven.

For some basic dependency analysis you can also use the command line, and do something like this:

$ find some_module/ -name *.java | \
    xargs cat $1 | \
    gawk 'match($0, /import (no.*);/, gr) {print gr[1]}' | \
    sort | uniq  | \
    grep -v no.ignore.this.namespace # -v option add strings to ignore

A short explanation for the script:

1. find all java files recursively (find command)
2. print them to console with the cat command
3. grab the output and extract import statements with some specific pattern with regex (gawk)
4. sort, extract unique values (sort, uniq)
5. add some namespaces to ignore some strings (grep with a -v option)

SonarQube

Last but not least, you might want to load your code for further analysis to SonarQube. This is an extremely powerful and free tool for doing the static analysis of your code. Normally, you would do that via a plug-in in your continuous integration (CI) software, like Jenkins. However, there might be some cases when you might want to load this data manually, via a command-line.

To load the code to SonarQube you will first need to add a property file in the root directory of each module. It might look like this:

# Contents of sonar-project.properties file:
sonar.projectKey=my:SomeFancyProject
sonar.projectName=My SomeFancyProject
sonar.projectVersion=1.0
sonar.sources=.
sonar.sourceEncoding=ISO-8859-1

Then you will need to install and run SonarQube Scanner from each module directory that contains sonar-project.properties file:

sonar-scanner

As a bonus feature, I might also suggest that if you don’t have time setting up SonarQube on a separate machine, but want to take a quick peek on how your project is doing, you can download and boot it up as a Docker image in no time, and later decide whether you want to create a dedicated machine for running SonarQube, or keep it as it is. Just remember that the database the SonarQube image uses out of the box should not be used for anything other than testing.

Bildegjenkjenning gjort enkelt. Slik gjenkjenner du en bestemt gjenstand ved hjelp av et kamera.

• Utfordringen
• Litt teori
• Hvor starter jeg?
• La oss kjøre bildegjenkjenning
• Bonus: Video

Utfordringen

Denne uken har jeg fått utfordringen å foreslå en løsning på følgende problem:

“Med en enkel datamaskin, som en Raspberry Pi, et kamera og et fuglebrett, lag et program som kan artsbestemme hvilke fugler som er innom brettet og spiser i løpet av en dag.”

Siden spørsmålet var av mer teoretisk art, bestemte jeg meg for å begrense dette innlegget til å introdusere byggeblokkene og gi en forklaring på hvordan du typisk kan bygge et slikt system.

Å gjenkjenne gjenstander er en ganske vanlig oppgave nå for tiden, og den har blitt løst på flere forskjellige måter, og med forskjellige tilnærminger. La oss se på hvordan dette virker.

TL;DR Det fins noen tradisjonelle måter å lage bilde- og gjenstandsgjenkjenning på, som i OpenCV, og det fins noen teknikker basert på Deep Learning, som i TensorFlow. Vil du vite mer? Fortsett å lese.

Litt teori

De mer tradisjonelle prinsippene bak OpenCV for å oppdage er godt beskrevet i dette blogginnlegget, så vel som i denne veiledningen for å gjenkjenne (hold dere fast!) katter på bilder.

Mens dette innlegget forklarer hvordan bildegjenkjenning virker i TensorFlow, som bruker en modell kalt dypt konvolverende nevrale nettverk, vil det også la deg trene opp din egen bildeklassifiserer, slik som OpenCV.

Hvor starter jeg?

I utgangspunktet har du to valg – du bør enten lære opp din egen modell eller finne en modell som har blitt lært opp av noen andre, og du vil kanskje alltid starte med å se etter en forhåndsopplært modell. Men jo mer spesifiserte bildeklassifiseringskravene dine er, jo høyere er sjansen for at du må lære opp din egen modell. Dette vil være det samme enten du velger OpenCV eller TensorFlow.

Siden vi vil kjøre modellen på en relativt enkel datamaskin, kan du vurdere å gjøre prosesseringen i skyen. Det bør imidlertid også være mulig å kjøre OpenCV og TensorFlow på den nyeste Raspberry Pi.

La oss kjøre bildegjenkjenning

Nå vet du litt om teorien. La oss så ta en rask titt på hvordan vi kan lære opp modellene i våre behov.

TL;DR Måten dette virker på, er at du viser modellen ganske mange bilder av en gjenstand, og en tilsvarende mengde bilder uten gjenstanden på.

La oss si at du vil bruke modellen til å gjenkjenne fugler utenfor huset ditt i Norge. Et godt utgangspunkt vil da være å få en liste over de typiske artene som du mest sannsynlig ser i hagen din, og samle så mange bilder av hver art som mulig.

Her er hva du kan gjøre:

1. Start med å finne en liste over norske fugler på Wikipedia, eller Store Norske Leksikon.
2. Søk på nettet etter bilder av hver fugleart. Det kan være du vil automatisere dette arbeidet, og sørg for at du plukker ut bilder med de riktige opphavsrettstillatelsene.
3. Bruk disse bildene i opplæringen av din modell.
4. Sett opp din Raspberry Pi med et kamera og fuglebrett, og gjør deg klar til å identifisere fuglearter. Du bør optimalisere programvaren slik at bildeklassifiseringen ikke skjer hele tiden, men kun når man ser en bevegelse.

Pro tip: Det kan også være litt utfordrende å ta gode bilder av våre fjærkledde venner, så sørg for at kameraet ditt er godt plassert, og materen er på et godt opplyst sted. Uten å forstyrre dyrelivet, så klart.

Lurer du på om lignende systemer har blitt implementert eller om det er mulig? Svaret er ja. Her er det noen lenker til inspirasjon for videre lesning:

• En fugleklassifiserer med tensorflow
• Trene opp en fugleklassifiserer med Tensorflow og TFLearn
• Kjøretøyklassifisering og -telling med OpenCV
• Blomstergjenkjenning i naturen (på dansk)
• Maskinlæring er morsomt! Del 3: Dypt konvolverende nevrale nettverk. (Ta også gjerne en titt på de andre delene i denne artikkelserien.)

Prøv nå å sette alt dette sammen, og fortell meg gjerne hvordan det går!

Lykke til!

Bonus: Video

En video vi har laget av mine tegninger og en kort forklaring på hvordan dette virker.