• The Existing State of Affairs
• The Moving Parts
• Conclusion

Let’s take a look into what we can do to achieve a better development environment than an average development project – a project that most of us have seen at some point in our professional lives, or maybe even are a part of right now. We will also look into some tools and patterns that will help us convert those projects into a paradise for the developers.

Just a few decades ago, we were working in ways that might look like unproductive, in the best case. Our development models were predominated by waterfalls, our IDEs were basic and we were compiling our projects by hand, using javac, or building up the CLASSPATH depending on the GOTO statements in a huge spaghetti code contained in countless bat files. Our code lived in a very simple versioning systems that were not distributed or supported branching strategies that are praised by the developers today. Our documentation lived in doc files on shared network drives, side by side with the simple issue tracking systems, that don’t even get close to what we have today.

Today, it is all different – we have Git, real issue tracking, IDEs, all that integrated with build servers and collaborative platforms. Yes, everything is much better, more effective and user-friendly, one might think that we are in the paradise already? Well… yes, things are fortunately getting better, however, we are still doing things in a way that might still give you nightmares, several decades from now.

Last years I have been working and invited to evaluate and help with an audit of various projects. Here are some of my observations and thoughts.

The Existing State of Affairs

This tweet describes it pretty well:

YOU ARE IN A LEGACY CODEBASE
> RUN TESTS
YOU HAVE NO TESTS
> READ SPEC
YOU HAVE NO SPEC
> WRITE FIX
YOU ARE EATEN BY AN ELDER CODE HACK.

Some of the issues are, naturally, remnants of the past – the legacy systems; but even those systems and most of the other problems we see today can be avoided if we slightly change our view at some of the main parts of the development process. In most of the cases, we would be aware of those issues, but we might need to explain and motivate the others – often people are responsible for the projects and those who prioritize the development and maintenance backlog.

The Moving Parts

The road to a great nightmare-free future consists of three components: the code quality, the development and build tools, and a good documentation and collaboration systems. When evaluating systems I often start asking some simple questions listed below to get an idea of the system.

1. The Code

The Code Quality

First thing off is the general code quality. I often start by asking about simple things – if the project has a coding standard, and if it is being followed. I also ask to take a quick peek at the code and check minor things like file encodings and MIME types. I also follow up with a question if the team is practicing code reviews, and how they are doing that.

While those things alone don’t have to mean anything, and are minor issues individually, together with other factors they still are initial indicators of possible neglect. This gives me a possibility to map areas where to look further.

In addition to that, there are also some more specific parts that will be listed as sub-sections below.

The third-party libraries

The role of the third party libraries and their use is often forgotten and neglected when considering code and system quality. This is quite unfortunate as this is the part of the code that you might not be able to patch easily, and is harder to maintain compared to your own codebase. Here are some simple questions that might help with getting a better grip on third-party libraries:

• Do you keep track of your third-party libraries?
• Do you regularly check if there are known issues or vulnerabilities in them?
• Do you have a plan for keeping them updated?
• Are the libraries you are using being actively maintained by the authors?
• Are the libraries you are using compatible with each other?
• Do the libraries you are using have appropriate licenses that are compatible with your system? (This also applies to the open source software licenses.)

Issues and vulnerabilities are being found and patched all the time. As an example for this, let me point to Google’s OSS-Fuzz Project that has found numerous security vulnerabilities in several critical open source projects. Unfortunately, even though many people are aware of the security issues in software in general, the library updates still often tend to be forgotten.

It is also worth noting that while most of the issues on my list above are security related, the last one might be of a legal sort, and probably is the most neglected of the issues listed.

The Architecture

I am often being asked to assess a system and tell something about its architecture compared to more modern systems. The different aspects of the system’s architecture will tell a lot about its maintainability both when it comes to further development, bug fixing, and keeping the system running. Some questions that might help with determining the state of the system would be:

• Does your architecture support automated deployment?
• Does your architecture support continuous deploy and delivery?
• Does your architecture support load balancing?
• Does your architecture support microservices?
• How is the architecture implemented in the code?

Tools for Maintaining the Code Quality

Some of this might sound familiar to you when you think of one or several systems you had been working with and you might now be wondering what you can do to improve the code quality? Further steps here would be starting to use proper tools that will be able to tell more about the various aspects of your code. Some of them can be used as plugins to your build system (like Maven), and some could be stand-alone tools.

Stand-alone tools for code analysis:

• SonarQube
• PMD
• FindBugs

Maven plugins to consider (more about plugins in my previous post):

• Assembly
• Versions
• Dependency
• Enforcer
• Surefire
• Failsafe
• Sonar
• Findbugs
• pmd

Bonus: See this post on command line tools for Java projects.

2. Development Tools and Strategies

Now, let’s talk about the development tools. All that fancy code and great architecture will not bring you any closer to a developer’s paradise if there will not be some proper tools to support the development. The code should live in a proper version control system that supports collaboration and things like branching and tagging. There should also be tools that help you with code quality analysis, static code analysis, etc. A good starting point here would be to start with answering the following about the project in question:

• Do you use a proper code versioning tool – Git, or even SVN?
• Do you have a branching (and tagging) strategy?
• Do you have a way of measuring code complexity?
• Do you have a way of measuring test coverage and results?
• Do you run static code analysis?

Some tools that can help you here (again, for Java-based systems):

• IDEs and IDE plug-ins that can do checks at commits, integrate with test and QA tools, etc.
• Build tools: Maven, Gradle, etc.
• Continuous integration tools: Jenkins, TeamCity, Bamboo, etc.
• Frameworks and tools for testing: to run unit tests, integration tests, UI tests, and end-to-end tests.

There are some further strategies and questions to consider:

• Are your environments easy to reproduce with minimal efforts – can you rebuild it by simply running a script?
• Do you have a proper pipeline from packaging, to delivery, to deploy?
• To what environments can you deploy automatically? With the same script, or command?
• Are your environments (like development, testing, staging, pre-prod, production) similar to each other?
• Do you follow the same process to deploy to each environment?
• Are QA and production running on physically separate hardware?
• Are you monitoring all of the environments? (i.e. are you able to see errors before they make it to QA or even production?)

3. Documentation and Collaboration Tools

Last, but not least, we will need to talk about the tools for collaboration and documentation. Without these tools, we will be back to the way things were several decades ago – with documents on shared network drives and other horrors of the 90’s that I mentioned at the beginning of this post. However, good wikis, other collaboration tools, and proper issue tracking will bring your software to another level, encouraging continuous improvement of the system.

• Wiki
• Collaboration – chat, etc.
• Issue tracking tools

No matter how obvious it might seem, it is still important to note that one should avoid multiple documentation and issue tracking systems. Unfortunately, even though it might sound obvious, it is more common than you think – I have seen my share of systems for documentation and issue tracking resulting in fragmented information and confusion.

Conclusion

There are several challenges connected with having and maintaining the good code quality. The first challenge is that a good code quality is not something you can achieve overnight. It takes time and energy to achieve that and it is a continuous process. You will need some tools, techniques, and methodology to prevail, and it will probably be easier to introduce all that from the beginning of a project.

The second challenge would be that it might be hard to convince the stakeholders of the project to invest time and resources into something that does not bring any visible improvements to the table – things like new features and bug fixes are more likely to get prioritized over something that cannot be easily measured.

Actually, while presenting on this topic at JavaOne 2017 in San Francisco, several of the attendees asked me about the ways of getting to a beautiful nightmare-free code and infrastructure, and the ways of convincing the stakeholders that this is the way to go. Unfortunately, there is no one simple solution to this, and the most valuable thing, in this case, would be to show the real value of the good quality code.

The measurements parameters to show the value can be:

• time it takes from the code is written to deploy,
• system stability,
• how often bugs are reported compared to earlier, or
• frequency of errors in logs.

So, what can you do as a developer on a project that might need some help, you might ask? You can just start by continuously suggesting improvements and showing their value to the customer, or the project manager. Now you just need to keep going and gradually improving the system, one small bit at a time.

• Introduction
• Directory Structure
• Code Metrics
• Encoding and MIME types
• Dependencies
• SonarQube

Introduction

This post will give you an overview of some command line tools that will be able to help you to get the feeling on how your project is doing. Most of the tools are widely available in the main Linux distributions and MacOS (some of them might need installation of Homebrew for Mac). On Windows it can be also made available via Cygwin, or Bash on Windows.

The commands below have been run and tested on a MacOS machine, and might sometimes need some slight modifications to be run on a Linux or a Windows machine. However, it should be able to give an idea of what it is possible to do with these tools. I will also be linking to the documentation for each of the programs in the post.

Directory Structure

Let’s start with something simple. Sometimes all you want to see is the contents and structure of the project without leaving the command line. The tree command might be able to help you out here.

It is highly customizable, takes lots of parameters, and is widely available for an installation via most of the package mangers. Command Prompt on Windows also contains a native alternative with the same name.

$ tree sourcecodefolder/ -L 1 -d
sourcecodefolder
├── src
├── tests
├── sql
└── docs

Code Metrics

If you want to see some metrics about the code in your project, like what languages that are used in the project, as well as information about number files, blank lines, comments, and lines of code, you might want to try the cloc command (it stands for Count Lines of Code).

Simply install the program and point it at the project directory, or a zip file:

$ cloc sourcecodefolder/

It will give you a result that might look something like this:

-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Java                            33           1226           1026           3017
Python                           4            327            337            888
Markdown                         1             11              0             28
SQL                             10             10             12            212
-------------------------------------------------------------------------------
SUM:                            48           1574           1375           4145
-------------------------------------------------------------------------------

Also, it is possible to show all the information by percent:

$ cloc --by-percent cmb sourcecode.zip

Encoding and MIME types

Files, within a single project, with different, or wrong, encoding might mean trouble with showing non-ASCII characters correctly. They might even give you compilation errors. Therefore, finding and marking those files as quickly as possible might help you manage your project.

While MIME types, on the other hand, are not as bad at causing trouble, the diversity of the MIME types among the same kind of files, like *.java files in this example, might indicate a lack of a standard for developer tools and code standard in the project.

This kind of information can be extracted with file command for each file, or it might be done a bit more automatic for the whole project, and it might look like this:

$ find . -name *.java | \
    xargs file -I $1 | \ 
    gawk 'match($0, /.* (.*); charset=(.*)$/, gr) {print gr[2]}' | \
    sort | uniq -c

A short explanation for the script:

1. find all java files recursively (find command)
2. process them with the file command
3. grab the output and extract data with regex (gawk)
4. sort, extract unique values and count (sort, uniq)

The result will look something like this:

 257 iso-8859-1
 117 unknown-8bit
 678 us-ascii

If you want to drill down a bit further and search for both MIME types and encoding, showing the distribution of encoding per MIME type, you can also extract that from the information supplied by the file command like this:

$ find . -name *.java | \ 
    xargs file -I $1 | \
    gawk 'match($0, /.* (.*); charset=(.*)$/, gr) \
    {print gr[1] " --- " gr[2]}' | \
    sort | uniq -c

A short explanation for the script is almost the same as above. The only difference is that we use information from both of the regex groups (gr[1], and gr[2]). The result will look something like this:

  15 text/html --- iso-8859-1
  34 text/html --- us-ascii
 134 text/plain --- iso-8859-1
  16 text/plain --- unknown-8bit
   1 text/plain --- us-ascii
   1 text/x-c --- iso-8859-1
   8 text/x-c --- us-ascii
   5 text/x-c++ --- iso-8859-1
   6 text/x-c++ --- us-ascii

Dependencies

Now, over to a bit more advanced stuff – analyzing dependencies in your project. Why would you want to do that? Well, in short: dependencies increase complexity, and cyclic dependencies are bad for your project and your health. They also hurt your modularity and complicate the build process.

This kind of analysis can be done with jdepend. It is easy to run and can be run both separately, and as a part of a build tool, like Maven.

For some basic dependency analysis you can also use the command line, and do something like this:

$ find some_module/ -name *.java | \
    xargs cat $1 | \
    gawk 'match($0, /import (no.*);/, gr) {print gr[1]}' | \
    sort | uniq  | \
    grep -v no.ignore.this.namespace # -v option add strings to ignore

A short explanation for the script:

1. find all java files recursively (find command)
2. print them to console with the cat command
3. grab the output and extract import statements with some specific pattern with regex (gawk)
4. sort, extract unique values (sort, uniq)
5. add some namespaces to ignore some strings (grep with a -v option)

SonarQube

Last but not least, you might want to load your code for further analysis to SonarQube. This is an extremely powerful and free tool for doing the static analysis of your code. Normally, you would do that via a plug-in in your continuous integration (CI) software, like Jenkins. However, there might be some cases when you might want to load this data manually, via a command-line.

To load the code to SonarQube you will first need to add a property file in the root directory of each module. It might look like this:

# Contents of sonar-project.properties file:
sonar.projectKey=my:SomeFancyProject
sonar.projectName=My SomeFancyProject
sonar.projectVersion=1.0
sonar.sources=.
sonar.sourceEncoding=ISO-8859-1

Then you will need to install and run SonarQube Scanner from each module directory that contains sonar-project.properties file:

sonar-scanner

As a bonus feature, I might also suggest that if you don’t have time setting up SonarQube on a separate machine, but want to take a quick peek on how your project is doing, you can download and boot it up as a Docker image in no time, and later decide whether you want to create a dedicated machine for running SonarQube, or keep it as it is. Just remember that the database the SonarQube image uses out of the box should not be used for anything other than testing.

• 1. Technical Aspects
• 2. Legal Aspects
• 3. Rapid Development
• 4. Documentation
• 5. Testing and QA
• Bonus: Video

Apache Maven is a build automation tool used primarily for Java projects. It has been around for a while, and it does not seem to be going anywhere anytime soon (whether some of you like it or not). Recently, it has been confirmed yet again, this time in ZeroTurnaround’s Java Tools and Technologies Landscape Report 2016 (see the Build Tools section).

The good thing about Maven is that you can do almost anything with it and its plugins. With such a great ecosystem of plugins, able to do nearly anything, one might wonder where to begin when setting up a new Maven project or improving your old one.

I usually like to think of five main categories – or pillars – that we will be looking into in this post. Each category will have a set of example plugins that are meant to serve merely as a starting point.

In this post, I will assume that you have some experience with build tools in general and Maven in particular.

1. Technical Aspects

First of all, you would like to make sure that all the technical stuff is in place. You might want to automate the packaging, the checks if the project is running the latest version of all artifacts, that all the dependencies are met, and that the unused dependencies are removed. You might even want to define your own rules for all the things that cause troubles, or just simply annoy you down the line, making the build fail if they are not met.

With all that automation in place, you will be getting predictable results and nice packaging from the first day of the project.

Some of the plugins that should be mentioned here:

• Versions plugin does the versions management of artifacts in a project’s POM file
• Dependency plugin helps you analyzing dependencies, building dependency trees, showing unused dependencies, etc
• Assembly plugin helps you packaging all the dependencies, modules, site documentation, and other files into a single distributable archive
• Enforcer plugin lets you make your own rules! You can set up your build to break if some of your requirements are not met

2. Legal Aspects

With all the technical stuff out of the way, we might want to make sure that the legal side is taken care of as well.

Yes, I know, it might be less fun thinking about licenses than writing code, but it is still something that has to be done. You still have to release your code under some kind of license, and you will have to make sure that the third-party licenses do not violate your own licensing. So, why not leave that job to a plugin?

The following plugin will manage the license of a maven project and its dependencies; it will also update file headers, download dependency licenses, check third-party licenses, etc.

• License plugin

3. Rapid Development

Now, back to coding. Or even better – to seeing the results of your hard work. The chances are that you will need some kind of web or application server for running your code, and you want to able to deploy to that server in no time. Another kind of plugins that will be helping you from the very first day of the project.

Some of the plugins that can be mentioned here:

• Apache Tomcat plugin
• Jetty plugin
• WildFly plugin
• …or any other deploy plugin, depending on your application

4. Documentation

Every decent project must also be properly documented. However, this is something that developers might postpone until the end. Well, no more! This kind of plugins will help you to get started early and will help you to produce some beautiful (maybe?) and maintainable docs.

You might even consider coupling these with some rules in the Enforcer plugin, but tread carefully as too many, and too strict rules can, and usually do, backfire.

• Site plugin
• Asciidoctor plugin

5. Testing and QA

By now, your code should be looking great, with all the right licenses, and up and running in no time. So, how about squashing some [virtual] bugs? The list below might help you setting up a proper QA environment and fixing bugs before the come crawling to your production servers.

• Surefire plugin
• Failsafe plugin
• SonarQube plugin
• FindBugs plugin
• PMD plugin

Bonus: Video

A video of a talk I gave about this topic at JavaZone in Oslo (in Norwegian).