• Intro
• 1. Before you begin
• 2. Create a Cloud Function
• 3. Deploy the Cloud Function

Intro

Recently we decided to migrate our builds from Travis CI to Google Cloud Build to speed up the builds. The process was quite easy and flawless; however, we were still missing a few minor things. One of them was the notifications from Cloud Build to our #ops channel in Slack. This was slightly annoying because you would not know if the build was finished and the site was deployed, or if it failed for some reason.

Integrating with Cloud Build was a bit more different than what you are used to from integrations with Jenkins or Travis CI. Normally you would just create a webhook that would call an interface in the Slack API. In Cloud Build, on the other hand, everything is getting posted to the Pub/Sub queue built into the platform, and here you would just need to subscribe to the specific queue and listen for the events. To achieve the latter, you would need a small serverless function to listen for these events and to call the Slack API.

Note that here we will, technically, be using paid services on Google Cloud Platform, as both Cloud Build, Cloud Pub/Sub, and Cloud Functions are billable components. However, since all the components above provide a generous free tier, you will need to work hard to get passed the free tier with this setup.

• Cloud Build: Free first 120 builds-minutes per day for Basic machine type (n1-standard-1).
• Cloud Pub/Sub: Free first 10GB per month (pricing).
• Cloud Functions: Free first 2 million invocations per month (pricing).

1. Before you begin

1.1 Prepare your GCP project

I assume you have a Google Cloud Account, and that you have signed in to your account.

1. Select or create a Google Cloud Platform project, e.g. from the Manage resources page.
2. Make sure that billing is enabled for your Google Cloud Platform project.
3. Enable the Cloud Functions and Cloud Pub/Sub. You can also enable the APIs using this link.
4. Use Cloud Shell right from the browser, or you can Install and initialize the Cloud SDK on your own machine.
5. If you have installed the Cloud SDK, update and install gcloud components:

gcloud components update &&
gcloud components install alpha beta

1.2 Prepare your Slack App

I assume you have Slack installed and that you have created and signed-in to your account.

Create a new Slack app:

1. Choose the app’s name and your Slack team. Click Create.
2. Click Incoming Webhooks.
3. Activate incoming webhooks.
4. Click Add New Webhook to Workspace. An authorization page opens.
5. From the drop-down menu, select the channel to which you would like notifications sent.
6. Click Authorize.
7. A webhook for your Slack application has been created. Copy the webhook URL and save it for later use.

2. Create a Cloud Function

We need to create a Cloud Storage bucket to stage your Cloud Functions files. Use [STAGING_BUCKET_NAME] that is a globally-unique bucket name (such as [PROJECT-ID]_cloudbuilds):

gsutil mb gs://[STAGING_BUCKET_NAME]

You should see the following output:

Creating gs://[PROJECT-ID]_cloudbuilds/[STAGING_BUCKET_NAME]...

Next, create a directory on your local system for the application code:

mkdir ~/gcb_slack
cd ~/gcb_slack

Then, create the following two files in the gcb_slack directory.

File 1: package.json

{
  "name": "google-container-slack",
  "version": "0.0.1",
  "description": "Slack integration for Google Cloud Build, using Google Cloud Functions",
  "main": "index.js",
  "dependencies": {
    "@slack/client": "4.10.0"
  }
}

File 2: index.js

Note: Make sure to update SLACK_WEBHOOK_URL in the code below.

const IncomingWebhook = require('@slack/client').IncomingWebhook;
const SLACK_WEBHOOK_URL = "<INSERT YOUR WEBHOOK FROM STEP 1.2>"

const webhook = new IncomingWebhook(SLACK_WEBHOOK_URL);

// subscribe is the main function called by Cloud Functions.
module.exports.subscribe = (event, callback) => {
 const build = eventToBuild(event.data.data);

  // Skip if the current status is not in the status list.
  // Add additional statues to list if you'd like:
  // QUEUED, WORKING, SUCCESS, FAILURE,
  // INTERNAL_ERROR, TIMEOUT, CANCELLED
  const status = ['SUCCESS', 'FAILURE', 'INTERNAL_ERROR', 'TIMEOUT'];
  if (status.indexOf(build.status) === -1) {
    return callback();
  }

  // Send message to Slack.
  const message = createSlackMessage(build);
  webhook.send(message, callback);
};

// eventToBuild transforms pubsub event message to a build object.
const eventToBuild = (data) => {
  return JSON.parse(new Buffer(data, 'base64').toString());
}

// createSlackMessage create a message from a build object.
const createSlackMessage = (build) => {
  let message = {
   text: `Build \`${build.id}\``,
    mrkdwn: true,
    attachments: [
      {
        title: 'Build logs - Your Custom Message Goes Here',
        title_link: build.logUrl,
        fields: [{
          title: 'Status',
          value: build.status
        }]
      }
    ]
  };
  return message
}

3. Deploy the Cloud Function

To deploy the subscribe function with a Cloud Pub/Sub trigger, run the following command in the gcb_slack directory:

gcloud functions deploy subscribe --stage-bucket [STAGING_BUCKET_NAME] \
    --trigger-topic cloud-builds

where [STAGING_BUCKET_NAME] is the name of your staging Cloud Storage Bucket that you defined earlier.

You should see an output confirming the creation of the cloud function and status: READY.

After you’ve completed deployment of the Cloud Function, when a build event occurs, you will receive a Slack notification.

Also, feel free to customize your app, like adding a custom icon, as I did with mine. ☝️

Containers are something that we use to run our applications and, normally, we dispose of the whole container when we build a new version of the application or need to upgrade something in the setup. This means that containers are generally having a short lifespan.

However, in this case, I want to show you how to build something that exists for an even shorter period of time and that can be used as an alternative to a local setup for building and testing applications locally before pushing it to test, staging, production, etc.

This is a simplified example of what is being done on a much bigger scale with moving your CI/CD pipelines to such disposable containers, and with libraries like Testcontainers.

In this case, I would like to show you how to setup Jekyll applications, but this can be easily applied to any kind of applications written in any of your favorite languages, like Java or Python. Until recently, I have been running a Jekyll installation locally with all dependencies installed on my machine. However, it has been a bit challenging when moving between machines and reinstalling operating systems. To simplify the process, I decided to containerize the local build and test processes.

I wanted the following:

• To build my code from and to the local folder on my (host) machine
• Run the application (in this case this blog) from a local folder on my (host) machine
• Avoid setting up the environment, or have a minimal and portable setup
• Avoid environment clean-up – I didn’t want to hold on to the unnecessary containers and container images

TL;DR: The solution

(see next section for the explanation)

$ export JEKYLL_VERSION=3.8
$ docker run --rm --volume="$PWD:/srv/jekyll" \
       -it jekyll/jekyll:$JEKYLL_VERSION jekyll build
$ docker run --name newblog --volume="$PWD:/srv/jekyll" -p 4000:4000 \
       -it jekyll/jekyll:$JEKYLL_VERSION jekyll serve --watch --drafts

Explanation – line by line

So, let’s take a closer look at each of the lines:

1: export JEKYLL_VERSION=3.8

Just setting up versions that will be used later – a bit of housekeeping. Nothing exciting here.

2: docker run --rm --volume="$PWD:/srv/jekyll" \
        -it jekyll/jekyll:$JEKYLL_VERSION jekyll build

Here, we build the code and output it to the same disk volume as the source code, i.e. the volume that is shared with my host machine. Now I have the built version on my machine without the hassle of setting up the local build environment. In addition to that, I will be doing some clean-up, by deleting the build container after the build job is finished.

• --rm – just execute the command and clean-up (remove the container, file system, etc.)
• --volume – mapping the current directory to /srv/jekyll in the container
• -it instructs Docker to allocate a pseudo-TTY connected to the container’s stdin; creating an interactive shell in the container
  • -i – attach container’s STDIN
  • -t – allocate a pseudo-TTY
• jekyll/jekyll:$JEKYLL_VERSION – Docker image to use and the tag
• jekyll build – command to run

3. docker run --name newblog --volume="$PWD:/srv/jekyll" -p 4000:4000 \
        -it jekyll/jekyll:$JEKYLL_VERSION jekyll serve --watch --drafts

This will create another container that will be running our application. Here we will need to add a few other parameters – like mapping the container ports to the ports on the local machine and giving the container a name.

• --name newblog – give your container a name
• --volume – mapping the current directory to /srv/jekyll in the container
• -p – bind port 4000 of the container to TCP port 4000 (-p host_machine:container)
• -it instructs Docker to allocate a pseudo-TTY connected to the container’s stdin; creating an interactive shell in the container
  • -i – attach container’s STDIN
  • -t – allocate a pseudo-TTY
• jekyll/jekyll:$JEKYLL_VERSION – Docker image to use and the tag
• jekyll serve --watch --drafts – command to run

Now you can stop the container with CTRL+c, and restart it again with:

$ docker start newblog -i

If you don’t want the container being persistent on your system, you can simply add --rm as in the previous command:

$ docker run --rm --name newblog --volume="$PWD:/srv/jekyll" -p 4000:4000 \
       -it jekyll/jekyll:$JEKYLL_VERSION jekyll serve --watch --drafts

• Moving Around the Command Line
• Editing Commands in the Command Line
• Bonus

Using the command line can simplify and even automate many of the operations we do on a computer. However, using the command line can mean quite a bit of typing and a possibly large number of parameters. In this post, I would like to focus on how to navigate the cursor and edit the command line, while leaving all the other Bash tricks for the future posts.

I also have created simple graphics to illustrate some of the main shortcuts listed below. This (hi-res) image can be printed for future reference.

Note: Please note that all commands containing ALT combinations might not work depending on your system configuration, and most definitely not work on MacOS. Normally, it is because these combinations are mapped to something else. However, you can still use the same shortcuts simply by replacing ALT with ESC.

Moving Around the Command Line

So, let’s first speak about how to move the cursor around – because using just arrow keys is often not the most optimal way of navigating. Sometimes you might want to go to the beginning of the line, to the end of the line, or simply jump from one word to another, where word – in this context – is set of characters separated by spaces (or sometimes other special characters), or as documentation states it:

A sequence of characters considered as a single unit by the shell. Also known as a token.

# Moving the cursor – fast
CTRL+a         Go to the beginning of the line (same as Home)
CTRL+e         Go to the End of the line (same as End)
ALT+b / ESC+b  Go one word back (to the left)
ALT+f / ESC+f  Go one word forward (to the right)

# Moving the cursor – one character at a time
CTRL+f         Go forward one character
CTRL+b         Go backward one character

# Using history
CTRL+r         Backwards search in previously executed commands (history)
CTRL+p         Previous command (same as Up arrow)
CTRL+n         Next command (same as Down arrow)

Editing Commands in the Command Line

Now that we are able to navigate freely along the command line, it is time to do some modifications. Here, we will see how to delete, cut, paste, and swap words and characters.

# Deleting whole words
ALT+Del        Delete the word before (to the left of) the cursor
ALT+d / ESC+d  Delete the word after (to the right of) the cursor
CTRL+w         Cut the word before the cursor to the clipboard

# Deleting parts of the line
CTRL+k         Cut the line after the cursor to the clipboard
CTRL+u         Cut/delete the line before the cursor to the clipboard

# Deleting single characters
CTRL+d         Delete character under the cursor (same as Delete key)
CTRL+h         Delete character before the cursor (same as Backspace key)

# Paste, Undo, revert, and more
CTRL+l         Clear the screen (similar to the 'clear' command)
CTRL+y         Paste the last thing to be cut (yank)
CTRL+_         Undo
ALT+r / ESC+r  Revert the changes and replace with the line as it was 
                in History.

# Swap 'em!
CTRL+t         Swap the last two characters before the cursor
ALT+t / ESC+t  Swap current word with previous
 
# Convert to UPPER, lower, or Sentence case
ALT+u / ESC+u  Capitalise characters from the cursor to the end of 
                the current word and move to the end of the word.
ALT+l / ESC+l  Lower the case of characters from the cursor to the
                end of the current word and move to the end of the word.
ALT+c / ESC+c  Capitalize the character under the cursor position 
                and move to the end of the word.

Bonus

First, the most obvious – you can always find more gems in the man pages for Bash both in your terminal and online (for instance on this mirror). To view it in your terminal, type:

$ man bash

Now, over to something different. Since we have been talking about the command line and shells it is worth mentioning some less-known (and sometimes “as a curiosity”) shortcuts in another terminal – Command Prompt, cmd.exe:

Function keys in cmd.exe:
  - F1: Pastes the last executed command (character by character)
  - F2: Pastes the last executed command (up to the entered character)
  - F3: Pastes the last executed command
  - F4: Deletes current prompt text up to the entered character
  - F5: Pastes recently executed commands (does not cycle)
  - F6: Pastes ^Z to the prompt
  - F7: Displays a selectable list of previously executed commands
  - F8: Pastes recently executed commands (cycles)
  - F9: Asks for the number of the command from the F7 list to paste

Good luck! Try them out and let me know how that goes!

• The Existing State of Affairs
• The Moving Parts
• Conclusion

Let’s take a look into what we can do to achieve a better development environment than an average development project – a project that most of us have seen at some point in our professional lives, or maybe even are a part of right now. We will also look into some tools and patterns that will help us convert those projects into a paradise for the developers.

Just a few decades ago, we were working in ways that might look like unproductive, in the best case. Our development models were predominated by waterfalls, our IDEs were basic and we were compiling our projects by hand, using javac, or building up the CLASSPATH depending on the GOTO statements in a huge spaghetti code contained in countless bat files. Our code lived in a very simple versioning systems that were not distributed or supported branching strategies that are praised by the developers today. Our documentation lived in doc files on shared network drives, side by side with the simple issue tracking systems, that don’t even get close to what we have today.

Today, it is all different – we have Git, real issue tracking, IDEs, all that integrated with build servers and collaborative platforms. Yes, everything is much better, more effective and user-friendly, one might think that we are in the paradise already? Well… yes, things are fortunately getting better, however, we are still doing things in a way that might still give you nightmares, several decades from now.

Last years I have been working and invited to evaluate and help with an audit of various projects. Here are some of my observations and thoughts.

The Existing State of Affairs

This tweet describes it pretty well:

YOU ARE IN A LEGACY CODEBASE
> RUN TESTS
YOU HAVE NO TESTS
> READ SPEC
YOU HAVE NO SPEC
> WRITE FIX
YOU ARE EATEN BY AN ELDER CODE HACK.

Some of the issues are, naturally, remnants of the past – the legacy systems; but even those systems and most of the other problems we see today can be avoided if we slightly change our view at some of the main parts of the development process. In most of the cases, we would be aware of those issues, but we might need to explain and motivate the others – often people are responsible for the projects and those who prioritize the development and maintenance backlog.

The Moving Parts

The road to a great nightmare-free future consists of three components: the code quality, the development and build tools, and a good documentation and collaboration systems. When evaluating systems I often start asking some simple questions listed below to get an idea of the system.

1. The Code

The Code Quality

First thing off is the general code quality. I often start by asking about simple things – if the project has a coding standard, and if it is being followed. I also ask to take a quick peek at the code and check minor things like file encodings and MIME types. I also follow up with a question if the team is practicing code reviews, and how they are doing that.

While those things alone don’t have to mean anything, and are minor issues individually, together with other factors they still are initial indicators of possible neglect. This gives me a possibility to map areas where to look further.

In addition to that, there are also some more specific parts that will be listed as sub-sections below.

The third-party libraries

The role of the third party libraries and their use is often forgotten and neglected when considering code and system quality. This is quite unfortunate as this is the part of the code that you might not be able to patch easily, and is harder to maintain compared to your own codebase. Here are some simple questions that might help with getting a better grip on third-party libraries:

• Do you keep track of your third-party libraries?
• Do you regularly check if there are known issues or vulnerabilities in them?
• Do you have a plan for keeping them updated?
• Are the libraries you are using being actively maintained by the authors?
• Are the libraries you are using compatible with each other?
• Do the libraries you are using have appropriate licenses that are compatible with your system? (This also applies to the open source software licenses.)

Issues and vulnerabilities are being found and patched all the time. As an example for this, let me point to Google’s OSS-Fuzz Project that has found numerous security vulnerabilities in several critical open source projects. Unfortunately, even though many people are aware of the security issues in software in general, the library updates still often tend to be forgotten.

It is also worth noting that while most of the issues on my list above are security related, the last one might be of a legal sort, and probably is the most neglected of the issues listed.

The Architecture

I am often being asked to assess a system and tell something about its architecture compared to more modern systems. The different aspects of the system’s architecture will tell a lot about its maintainability both when it comes to further development, bug fixing, and keeping the system running. Some questions that might help with determining the state of the system would be:

• Does your architecture support automated deployment?
• Does your architecture support continuous deploy and delivery?
• Does your architecture support load balancing?
• Does your architecture support microservices?
• How is the architecture implemented in the code?

Tools for Maintaining the Code Quality

Some of this might sound familiar to you when you think of one or several systems you had been working with and you might now be wondering what you can do to improve the code quality? Further steps here would be starting to use proper tools that will be able to tell more about the various aspects of your code. Some of them can be used as plugins to your build system (like Maven), and some could be stand-alone tools.

Stand-alone tools for code analysis:

• SonarQube
• PMD
• FindBugs

Maven plugins to consider (more about plugins in my previous post):

• Assembly
• Versions
• Dependency
• Enforcer
• Surefire
• Failsafe
• Sonar
• Findbugs
• pmd

Bonus: See this post on command line tools for Java projects.

2. Development Tools and Strategies

Now, let’s talk about the development tools. All that fancy code and great architecture will not bring you any closer to a developer’s paradise if there will not be some proper tools to support the development. The code should live in a proper version control system that supports collaboration and things like branching and tagging. There should also be tools that help you with code quality analysis, static code analysis, etc. A good starting point here would be to start with answering the following about the project in question:

• Do you use a proper code versioning tool – Git, or even SVN?
• Do you have a branching (and tagging) strategy?
• Do you have a way of measuring code complexity?
• Do you have a way of measuring test coverage and results?
• Do you run static code analysis?

Some tools that can help you here (again, for Java-based systems):

• IDEs and IDE plug-ins that can do checks at commits, integrate with test and QA tools, etc.
• Build tools: Maven, Gradle, etc.
• Continuous integration tools: Jenkins, TeamCity, Bamboo, etc.
• Frameworks and tools for testing: to run unit tests, integration tests, UI tests, and end-to-end tests.

There are some further strategies and questions to consider:

• Are your environments easy to reproduce with minimal efforts – can you rebuild it by simply running a script?
• Do you have a proper pipeline from packaging, to delivery, to deploy?
• To what environments can you deploy automatically? With the same script, or command?
• Are your environments (like development, testing, staging, pre-prod, production) similar to each other?
• Do you follow the same process to deploy to each environment?
• Are QA and production running on physically separate hardware?
• Are you monitoring all of the environments? (i.e. are you able to see errors before they make it to QA or even production?)

3. Documentation and Collaboration Tools

Last, but not least, we will need to talk about the tools for collaboration and documentation. Without these tools, we will be back to the way things were several decades ago – with documents on shared network drives and other horrors of the 90’s that I mentioned at the beginning of this post. However, good wikis, other collaboration tools, and proper issue tracking will bring your software to another level, encouraging continuous improvement of the system.

• Wiki
• Collaboration – chat, etc.
• Issue tracking tools

No matter how obvious it might seem, it is still important to note that one should avoid multiple documentation and issue tracking systems. Unfortunately, even though it might sound obvious, it is more common than you think – I have seen my share of systems for documentation and issue tracking resulting in fragmented information and confusion.

Conclusion

There are several challenges connected with having and maintaining the good code quality. The first challenge is that a good code quality is not something you can achieve overnight. It takes time and energy to achieve that and it is a continuous process. You will need some tools, techniques, and methodology to prevail, and it will probably be easier to introduce all that from the beginning of a project.

The second challenge would be that it might be hard to convince the stakeholders of the project to invest time and resources into something that does not bring any visible improvements to the table – things like new features and bug fixes are more likely to get prioritized over something that cannot be easily measured.

Actually, while presenting on this topic at JavaOne 2017 in San Francisco, several of the attendees asked me about the ways of getting to a beautiful nightmare-free code and infrastructure, and the ways of convincing the stakeholders that this is the way to go. Unfortunately, there is no one simple solution to this, and the most valuable thing, in this case, would be to show the real value of the good quality code.

The measurements parameters to show the value can be:

• time it takes from the code is written to deploy,
• system stability,
• how often bugs are reported compared to earlier, or
• frequency of errors in logs.

So, what can you do as a developer on a project that might need some help, you might ask? You can just start by continuously suggesting improvements and showing their value to the customer, or the project manager. Now you just need to keep going and gradually improving the system, one small bit at a time.

• Getting Started
• Commands, Files, and Folders Inside a Container
• Cleanup

In this post, I decided to share some of the basic commands you might need to get started with Docker. This is neither an extensive list of the commands available, nor all of the commands you might need. This is merely me sharing a prettified list of my cheat sheet for Docker basics with you.

Getting Started

Before we get started, it might be a good idea to note that all of the commands below are written without sudo. If your installation is not running without sudo (assuming that you are running Linux), you might want to check out the post-installation guide for Linux in the Docker docs.

1. Check if Everything Works

First things first, you can use this simple command to check that your installation is fine. Note: Make sure you have right CPU architecture for your images. Raspberry Pi (ARM) things will not run on x86 architecture, and vice versa.

For x86:

$ docker run docker/whalesay cowsay Hello World!

For Raspberry Pi / AMD:

$ docker run -d -p 80:80 hypriot/rpi-busybox-httpd

2. List Containers

After creating containers, first thing you might want to do is to see what containers you have up and running. To list all running containers you can use:

$ docker ps

This command will give you a list similar to this:

CONTAINER ID        IMAGE                       COMMAND                  CREATED             STATUS                    PORTS                NAMES
e85753d57a67        easypi/dokuwiki-arm         "/bin/sh -c 'php-f..."   1 days ago          Up 23 hours               0.0.0.0:80->80/tcp   mywiki

However, it will not show you any stopped containers. To list all local containers use the -a option:

$ docker ps -a

The output will be more like this (note that is shows also stopped, or even failed containers):

CONTAINER ID        IMAGE                       COMMAND                  CREATED             STATUS                    PORTS                NAMES
573193cf1d5e        hypriot/rpi-busybox-httpd   "/bin/busybox http..."   2 days ago          Exited (0) 5 hours ago                         mytest
e85753d57a67        easypi/dokuwiki-arm         "/bin/sh -c 'php-f..."   1 days ago          Up 23 hours               0.0.0.0:80->80/tcp   mywiki

More on docker ps in the Docker docs.

3. List Images

To list all the images available on your system, simply do this:

$ docker images

4. Containers vs. Images?

What is the difference between containers and images, you might wonder? Well, I have a link for you. This will hopefully help you to understand how Docker manages the data within your images and containers.

5. Starting and Stopping Containers

Another two basic commands – starting and stopping containers:

$ docker start <container_id>
$ docker stop <container_id>

Note: The docker run command first creates a writeable container layer over the specified image, and then starts it using the specified command. That is, docker run is equivalent to the API’s /containers/create, and then /containers/<id>/start.

Commands, Files, and Folders Inside a Container

6. Run Any Command from a Container

You can run any command in a running container just knowing its ID (or name):

$ docker exec -it <container_id_or_name> echo "Hello from container!"

7. Getting Into Containers

Since you can run any command, then you can (obviously) also run a shell from a container; if you have any. This will be a bit similar to running an ssh command to connect remotely to a regular Linux box (given you have bash or sh in the container):

$ docker exec -it <container_id_or_name> bash
$ # or:
$ docker exec -it <container_id_or_name> sh

8. Copy Files From and To Containers

Another useful trick you might need is to copy some files to and from a container. Your friend here is the docker cp command (link to the docs):

$ # To container:
$ docker cp foo.txt <container_name>:/foo.txt
$ # From container:
$ docker cp <container_name>:/foo.txt foo.txt

Cleanup

After playing round with all the images and containers, you might realize that you have quite a collection of these on your drive, just taking up space.

9. Remove Containers

To remove the unused or unwanted containers, you can run the docker rm command with the IDs of those images. The IDs can be retrieved with the docker ps -a command, mentioned above.

$ docker rm <container_id>

10. Remove Images

The docker rmi command followed by the IDs of images will help you to remove the unused or unwanted images. The abovementioned docker images command will help you finding the correct IDs for the images in question.

$ docker rmi <container_id>

Have fun!

• Introduction
• Create the Service File
• Activate the Service

Introduction

After your Docker containers are set up and running, you might need to be able to start some of them automatically on a reboot or a crash. There are several ways of getting this done.

One of them is to use restart policies provided by Docker. They can be set to control whether your containers start automatically when they exit, or when Docker restarts.

Alternatively, you can use a process manager such as upstart, systemd, or supervisor instead. In this post, I want to show you how it is done with systemd.

Create the Service File

To create a service file that will be used by systemd (systemctl command), we will first need to get your container name. This can be done by running the following command in your shell:

$ docker ps -a

The output will look something like this. Select the right container from the list, and note its name in the last column. In this example, we will be using mywiki container.

CONTAINER ID        IMAGE                       COMMAND                  CREATED             STATUS                    PORTS                NAMES
573193cf1d5e        hypriot/rpi-busybox-httpd   "/bin/busybox http..."   2 days ago          Exited (0) 5 hours ago                         mytest
e85753d57a67        easypi/dokuwiki-arm         "/bin/sh -c 'php-f..."   1 days ago          Up 23 hours               0.0.0.0:80->80/tcp   mywiki

Now, we will need to create a file (choose an appropriate file name for the service):

$ sudo nano /etc/systemd/system/docker-dokuwiki.service

Paste the following into the file. Set a proper Description, and make sure to update the container name in ExecStart and ExecStop:

[Unit]
Description=DokuWiki Container
Requires=docker.service
After=docker.service

[Service]
Restart=always
ExecStart=/usr/bin/docker start -a mywiki
ExecStop=/usr/bin/docker stop -t 2 mywiki

[Install]
WantedBy=local.target

A couple of notes about the script above:

1. This file is called a unit file for systemd.
2. Make sure you don’t have any extra line brakes within the sections, like Unit, or Service.
3. The -a option in the Docker command for ExecStart makes sure it is running in attached mode, i.e., attaching STDOUT/STDERR and forwarding signals.
4. The -t option in the Docker command for ExecStop specifies seconds to wait for it to stop before killing the container.

Activate the Service

Before we can activate the service we have created, we need to reload the unit file. You will also need to run this command anytime you do any modifications to the unit files:

$ sudo systemctl daemon-reload

To activate the service run the following commands (remember to change the service name):

$ sudo systemctl start docker-dokuwiki.service
$ sudo systemctl enable docker-dokuwiki.service

To disable the service run the following commands (remember to change the service name):

$ sudo systemctl stop docker-dokuwiki.service
$ sudo systemctl disable docker-dokuwiki.service

Changes will come to effect on a reboot:

$ sudo reboot

Now you should have a container that will start on a server reboot, Docker restart, or a crash. Congratulations!

As a next step, you might want to look at (external documentation links):

• Adding some more parameters to the unit file.
• Available docker start options.
• Available docker start options.

• Posting via Confluence REST API
• Creating a Lambda Function in AWS
• Schedule it!

Some mundane tasks can easily be automated. In my case, I needed something that could post to our Confluence instance at a specific time every week. This is because we use blog posts in Confluence for announcing our weekly volunteer meetings and as a simple attendance management.

Posting via Confluence REST API

Confluence, a team collaboration software written in Java and quite often used in corporate environments, offers a REST API that makes it possible to perform many operations on the content. In this case, I was interested in posting new content with some text, and I was getting tired of doing that manually every week. So, I thought: “I spend a lot of time on this task. I should write a program automating it!”.

The first take on that problem was a simple Python script:

import json
import requests

url = "https://some-confluence-url.com/rest/api/content/"
headers = { "content-type": "application/json" }
payload = { "type": "blogpost",
            "title": "Fancy meeting title",
            "space": {
                "key": "mySpaceNameGoesHere"
            },
            "body": {
                "storage": {
                    "representation": "storage",
                    "value": "Meeting description goes here."
                }
            }
            #"history": {
            #    "createdDate": '2017-01-10T14:00:00.000Z'
            #},

        }

response = requests.post(url, 
                         data=json.dumps(payload), 
                         headers=headers, 
                         auth=("myUserName", "mySecretPassword"))
print json.dumps(response.json(), indent=4, sort_keys=True)

You should be able to run this script against your end-point and post your content. Just note that I used requests library here that needs to be installed separately, for instance, by using pip:

sudo pip install requests

Also, you will need to update the following values in the script:

• url – the URL to your site
• type – can be a blogpost or a page
• title – page or post title
• space – a Confluence space where the post will belong
• value – content of the page, can contain HTML tags
• auth – username and password (please note that, for security reasons, it might be a good idea to create a separate user with restricted access rights just for posting specific content)
• createdDate – you can also add the commented-out part to add a specific blog post date

The script uses basic authentication to post the content and, if successful, returns metadata about the post as a response. If it fails, the response from Confluence end-point should point you in the right direction.

Creating a Lambda Function in AWS

Now we are one step closer to automation. The script works, and we can post content with a simple push of a button. What now?

Well, now I needed something to run that script and something to trigger that action. The trigger is the time as I wanted to run our script weekly, but first I needed to deploy our script to the Cloud. Scheduling and triggers would come later.

Since we had some infrastructure on AWS already, I decided to deploy the script on the same platform. I wanted to deploy it as a simple stand-alone function in the Cloud – in AWS world it is called a Lambda function.

I started by wrapping our code into a function and making it AWS Lambda function compatible – to define a function that takes two arguments: event and context. I choose to call the function blogpost_handler:

#!/usr/bin/python
# -*- coding: utf-8 -*-

# File name: poster.py

import requests
import json
import datetime

def blogpost_handler(event, context): 

    # Script is run on Sunday, need to use date for Tuesday
    next_tue = (datetime.datetime.now()
                    +datetime.timedelta(days=2)).strftime("%Y-%m-%d")

    url = "https://some-confluence-url.com/rest/api/content/"
    headers = { "content-type": "application/json" }
    payload = { "type": "blogpost",
                "title": "Meeting: "+next_tue,
                "space": {
                    "key": "mySpaceNameGoesHere"
                },
                "body": {
                    "storage": {
                        "representation": "storage",
                        "value": "Meeting description goes here."
                    }
                }
            }

    
    response = requests.post(url, 
                             data=json.dumps(payload), 
                             headers=headers, 
                             auth=("myUserName", "mySecretPassword"))
    
    print json.dumps(response.json(), indent=4, sort_keys=True)

    return response.json()

Now, I needed to package the code to make it ready to deploy:

# 1. create folder
mkdir blogposter
cd blogposter

# 2. create and copy the code above into poster.py
touch poster.py

# 3. install requests package locally
pip install requests -t .

# 4. zip contents of the folder and create a zip file in a parent directory
zip -r9 ../pyposter.zip *

And we are ready to deploy to AWS! Start with creating a new Lambda Function and selecting a blank function for this job.

I didn’t choose any triggers for now and continued to the configuration page. When presented with function configuration you need to give it a name, description, runtime, and a .zip file with the code. At the end, you will need to define a handler (which will typically be filename.function_name), and a role.

At the end of the wizard, you can run it manually by pressing the Test button. If you did all the steps above, this should now create a new post on your Confluence. Congratulations, you have now set up a stand-alone function in the Cloud!

Schedule it!

Now that I have automated the task and deployed it to the Cloud, I needed something to trigger that function. In this case, I wanted to schedule it to run weekly, so function was set to be triggered by the CloudWatch events. It offers a good support for scheduling Lambda expressions.

I defined event source as a cron expression, and a target as a Lambda function I created in the previous step.

# 9:05, every Sunday
5 9 ? * 1 *

Now you (and I) have a function that is scheduled to run weekly and does not require a dedicated machine to run. It also has some good options for logging and sending alarms in case it fails.

• 1. Technical Aspects
• 2. Legal Aspects
• 3. Rapid Development
• 4. Documentation
• 5. Testing and QA
• Bonus: Video

Apache Maven is a build automation tool used primarily for Java projects. It has been around for a while, and it does not seem to be going anywhere anytime soon (whether some of you like it or not). Recently, it has been confirmed yet again, this time in ZeroTurnaround’s Java Tools and Technologies Landscape Report 2016 (see the Build Tools section).

The good thing about Maven is that you can do almost anything with it and its plugins. With such a great ecosystem of plugins, able to do nearly anything, one might wonder where to begin when setting up a new Maven project or improving your old one.

I usually like to think of five main categories – or pillars – that we will be looking into in this post. Each category will have a set of example plugins that are meant to serve merely as a starting point.

In this post, I will assume that you have some experience with build tools in general and Maven in particular.

1. Technical Aspects

First of all, you would like to make sure that all the technical stuff is in place. You might want to automate the packaging, the checks if the project is running the latest version of all artifacts, that all the dependencies are met, and that the unused dependencies are removed. You might even want to define your own rules for all the things that cause troubles, or just simply annoy you down the line, making the build fail if they are not met.

With all that automation in place, you will be getting predictable results and nice packaging from the first day of the project.

Some of the plugins that should be mentioned here:

• Versions plugin does the versions management of artifacts in a project’s POM file
• Dependency plugin helps you analyzing dependencies, building dependency trees, showing unused dependencies, etc
• Assembly plugin helps you packaging all the dependencies, modules, site documentation, and other files into a single distributable archive
• Enforcer plugin lets you make your own rules! You can set up your build to break if some of your requirements are not met

2. Legal Aspects

With all the technical stuff out of the way, we might want to make sure that the legal side is taken care of as well.

Yes, I know, it might be less fun thinking about licenses than writing code, but it is still something that has to be done. You still have to release your code under some kind of license, and you will have to make sure that the third-party licenses do not violate your own licensing. So, why not leave that job to a plugin?

The following plugin will manage the license of a maven project and its dependencies; it will also update file headers, download dependency licenses, check third-party licenses, etc.

• License plugin

3. Rapid Development

Now, back to coding. Or even better – to seeing the results of your hard work. The chances are that you will need some kind of web or application server for running your code, and you want to able to deploy to that server in no time. Another kind of plugins that will be helping you from the very first day of the project.

Some of the plugins that can be mentioned here:

• Apache Tomcat plugin
• Jetty plugin
• WildFly plugin
• …or any other deploy plugin, depending on your application

4. Documentation

Every decent project must also be properly documented. However, this is something that developers might postpone until the end. Well, no more! This kind of plugins will help you to get started early and will help you to produce some beautiful (maybe?) and maintainable docs.

You might even consider coupling these with some rules in the Enforcer plugin, but tread carefully as too many, and too strict rules can, and usually do, backfire.

• Site plugin
• Asciidoctor plugin

5. Testing and QA

By now, your code should be looking great, with all the right licenses, and up and running in no time. So, how about squashing some [virtual] bugs? The list below might help you setting up a proper QA environment and fixing bugs before the come crawling to your production servers.

• Surefire plugin
• Failsafe plugin
• SonarQube plugin
• FindBugs plugin
• PMD plugin

Bonus: Video

A video of a talk I gave about this topic at JavaZone in Oslo (in Norwegian).

• What’s going on?
• Setting the date in the front matter
• Setting the time zone in the Jekyll configuration file

What’s going on?

So, you have written a brand new post. You have run tests on a local Jekyll server, and everything looks great. You finally commit the changes to your GitHub site repository and push the changes to the server. You are excited and full of anticipation to see your new blog post online, you hit refresh in your browser, and… nothing. Another refresh – still nothing.

One of the reasons for this is that you have specified a date/time for your post in the YAML front matter in the header.

By setting a date in the front matter, you can override the date/time assigned to the post in the format YYYY-MM-DD HH:MM:SS. However, this notation will use the local time zone, set by your operating system. This works fine when running a local installation of Jekyll. However, if your Jekyll installation is running on a remote server (e.g., on the GitHub Pages server) the time on that server is likely not to be the same as the time on your local machine. Jekyll will then wait before processing the contents of the post until the specified time hits the clock on the server.

For your post to be processed according to your time zone, you can either add a UTC offset for your time zone for the post in question or specify a time zone in the Jekyll configuration file.

Setting the date in the front matter

To specify the UTC offset for each post use the following format: YYYY-MM-DD HH:MM:SS +/-TTTT, e.g. 2016-06-12 11:04:00 +0200. This might be useful if you want to specify the offset for some specific posts.

---
title: "Your title"
layout: post
date: 2016-06-12 11:04 +0200
---
Your content.

Setting the time zone in the Jekyll configuration file

If you want an offset to be applied to all of the dates on your site, you should configure Jekyll by setting the time zone for site generation in _config.yml.

Any entry from the IANA Time Zone Database is a valid value, e.g., Europe/Oslo. Wikipedia also has a list of all available values.

# set timezone
timezone: Europe/Oslo

If not specified, Jekyll will use the local time zone, as set by your operating system.