Amazon DynamoDB

Amazon DynamoDB is a key-value and document database that delivers single-digit millisecond performance at any scale. It’s a fully managed, multi-region, multi-active, durable database with built-in security, backup and restore, and in-memory caching for internet-scale applications. DynamoDB can handle more than 10 trillion requests per day and can support peaks of more than 20 million requests per second.

This post aims to help developers and operations team to understand the strengths and weaknesses of DynamoDB.

Data Modeling

DynamoDB supports a document oriented data model. To create a table, we just define the primary / partition key. Items can be added into these tables with a dynamic set of attributes similar to MongoDB. Items in DynamoDB correspond to rows and attributes correspond to columns in RDBMS. DynamoDB supports these data types - Number, String, Binary, Boolean, Set, List, Map.

Operations Ease

As it’s a managed service from Amazon, users are abstracted away from the underlying infrastructure and interact only with the database over a remote endpoint. There is no need to worry about operational concerns such as hardware, setup/configuration, throughput capacity planning, replication, software patching, or cluster scaling — making it very easy to get started.

In fact, there is no way to access the underlying infrastructure components such as the instances or disks. DynamoDB tables require users to reserve read capacity units (RCUs) and write capacity units (WCUs) upfront. Users are charged by the hour for the throughput capacity reserved (whether or not these tables are receiving any reads or writes).

Linear Scalability

DynamoDB supports auto sharding and load-balancing. This allows applications to transparently store ever-growing amounts of data. The linear scalability of DynamoDB is good for applications that need to handle growing datasets and IOPS requirements. However, this linear scalability comes with extreme costs beyond a certain point.

Amazon Ecosystem Integration

DynamoDB is well integrated into the AWS ecosystem. It means that end users do not need to figure out how to perform various integrations by themselves. Below are couple of examples of these integrations:

• Data can easily and cost-effectively be backed up to S3
• Security and access control is integrated into AWS IAM

Cost Effectiveness

DynamoDB’s pricing model can easily make it the single most expensive AWS service for a fast growing data set. Here are some reasons:

Higher provisioning to handle partitions

In DynamoDB, the total provisioned IOPS is evenly divided across all the partitions. Therefore, it is extremely important to choose a partition key that will evenly distribute reads and writes across these partitions.

Cost explodes for fast growing data sets

As data grows, so do the number of partitions in order to automatically scale out the data (each partition is a maximum of 10GB). However, the total provisioned throughput for a table does not increase. Thus, the throughput available for each partition will constantly decrease with data growth.

Indexes will result in additional cost

Applications wanting to query data on attributes that are not a part of the primary key need to create secondary indexes. Local Secondary Indexes do not incur extra cost, but Global Secondary Indexes require additional read and write capacity provisioned leads to additional cost.

Additional cost for caching tier

Applications wanting less latency, we should add cache to increase the performance. The caching tier DAX or Elastic Cache is an additional expense on the top of the database tier.

The ideal workloads for DynamoDB should have the following characteristics:

• Low write throughput.
• Small and constant dataset size, doesn’t have unknown data growth.
• Constant or predictable read throughput, should not be explosion or unpredictable.
• Applications that can tolerate eventual consistent reads, the least expensive data access operation in DynamoDB.

Some guidelines if you are going to use DynamoDB.

• Use GUID’s or Unique Attributes, instead of incremental IDs.
• Don’t try to normalise your tables.
• Keeping pre-computed data upon updates is efficient with DynamoDB if you need to query them often.
• Don’t try to keep many relationships across tables. This will end up needing to query multiple tables to retrieve required attributes.
• Design your tables, attributes, and indexes thinking of the nature of queries.
• Think about item sizes and using indexes effectively when listing items to minimise throughput requirements.
• Avoid using DynamoDB Scan operation whenever possible.

Read More

SLF4J vs LOG4J

Image Source: https://www.educba.com/

Do I go for SLF4j or LOG4j or both? Classic question from my team.

SLF4j has been around for sometime now and has been adopted heavily across. But certain things never get over. Hence I thought of writing down my thoughts as a blog to help others.

Going back to the question; SLF4j or LOG4j ? I think this question itself incorrect. SLF4j and LOG4j focus on different areas and they are meant to do two different things. It's like comparing apples and oranges.

SLF4j is a logging facade. It doesn’t do logging by itself and depends on a logging component like LOG4j, Logback or JLogging. SLF4j is an API designed to give generic access to many logging frameworks. So your log code within the application level remains same but the underlying logging framework can be switched without any kind of actual source code changes.

Once you get used to the syntax of SLF4j, then you don’t need to worry about syntax for other logging frameworks. Another major feature of SLF4j which convinced me to use it over my long time favourite LOG4j, is presence of placeholder, which is represented as {} in code. Placeholder is pretty much same as %s in format() method of String, because it gets substituted by actual string supplied at runtime. This not only reduces the number of string concatenations in your code, but also cost of creating string objects. Since Strings are immutable and they are created in String pool, they consume heap memory and most of the time they are not needed e.g. a String used in DEBUG statement is not needed when your application is running on ERROR level in production.

By using SLF4j, you can defer String creation at the runtime, which means only required Strings will be created. If you have been using LOG4j then you must be already familiar with a workaround of putting debug statement inside if() condition, but SLF4j placeholders are much better than that.

LOG4j Style:

if (LOGGER.isDebugEnabled()) {
                LOGGER.debug("Initiating Batch Processing... RequestId: " + requestId + ", Region: " + region);
}

SLF4j Style:

LOGGER.debug("Initiating Batch Processing... RequestId: {}, Region: {}", requestId, region);

You might be thinking what if I have multiple parameters? Well you can either use variable arguments version of log methods or pass them as Object array. It is very convenient and efficient way of logging. Remember, before generating final String for logging the message, this method checks if a particular log level is enabled or not, which not only reduces memory consumption but also CPU time involved in executing those String concatenation instruction in advance. It’s also worth knowing that logging has severe impact on performance of the application, and it’s always advised to have only mandatory logging in production environment.

Code Snippet from org.slf4j.impl.Log4jLoggerAdapter:

public void debug(String format, Object arg1, Object arg2) {
        if (logger.isDebugEnabled()) {
        FormattingTuple ft = MessageFormatter.format(format, arg1, arg2);
        logger.log(FQCN, Level.DEBUG, ft.getMessage(), ft.getThrowable());
        }
  }

Summary:

• SLF4j provides place holder based logging, which improves readability of code by removing checks like isDebugEnabled(), isInfoEnabled() etc.
• By using SLF4j logging method, you defer cost of constructing logging messages (String), which is both memory and CPU efficient.
• On a side note, less number of strings means less work for Garbage Collector; in turn better throughput and performance!
• Using SLF4j in your source will make it independent of any particular logging implementation i.e., no need to manage multiple logging configurations for multiple libraries.

So essentially, SLF4j does not replace LOG4j; they work together, hand in hand. SLF4j removes the dependency on LOG4j from your application and makes it easy to replace it in future with more capable library without any kind of source code changes.

Read More

MyBatis Database Migration Tool

Evolving databases has been one of the major challenges for software development. Often regardless of our software development methodology, the database follows a different change management process. The tools of the past have been GUI centric, proprietary for a particular database and/or carried a steep license cost. Yet, at the end of the day they suffered from the same challenges.

The MyBatis Migration Maven plugin is a simple command line tool that helps us to manage the database schema changes in a more systematic way where manual work can be avoided. It can be used for any relational database systems like postgres, mysql, oracle etc.

For migration, you create the sql script files as usual which will contain one more sql statements and save the file with a specific naming convention provided by the MyBatis Miration. File naming convention YYYYMMDDHHMMSS_file_name.sql. The timestamp provided in the filename should be unique otherwise the migration will abort with an error.

• Eg:- 20170506163015_create_member_details.sql

One of the good thing about MyBatis Migration is that the script file will contain sql statements to make the changes as well as to rever those changes. Hence the script file has been divided into two sections like below

• --// First migration. -- Migration SQL that makes the change goes here. 
• --//@UNDO -- SQL to undo the change goes here.

Here are the steps to integrate your new or an existing project with MyBatis Migration.

Add Maven Plugin and Dependency

<plugin>
         <groupId>org.mybatis.maven</groupId>
         <artifactId>migrations-maven-plugin</artifactId>
         <version>1.1.2</version>
         <configuration>
             <repository>src/main/resources/db/</repository>
             <output>dist/migration-name.sql</output>
         </configuration>
             <dependencies>
                 <dependency>
                     <groupId>org.postgresql</groupId>
                     <artifactId>postgresql</artifactId>
                     <version>9.4-1205-jdbc4</version>

                 </dependency>
             </dependencies>
     </plugin>

Add a property file

Under your src/main/resources add a sub-directory named environments. And create a property file and name it say development.properties. You can also create test.properties and production.properties files. The environment can be specified when running a migration by using the –env=<environment> option (without the path or “.properties” part). Below is the template which should be used for the property file.

## Base time zone to ensure times are consistent across machines
time_zone=GMT+0:00

## The character set that scripts are encoded with
# script_char_set=UTF-8

## JDBC connection properties.
driver=
url=
username=
password=

# Name of the table that tracks changes to the database
changelog=db_changelog

# If set to true, each statement is isolated in its own transaction.
# Otherwise the entire script is executed in one transaction.
auto_commit=false

# This controls how statements are delimited. 
# By default statements are delimited by an end of line semicolon. 
# Some databases may (e.g. MS SQL Server) may require a full line delimiter such as GO.
delimiter=;
full_line_delimiter=false

# This ignores the line delimiters and simply sends the entire script at once.
# Use with JDBC drivers that can accept large blocks of delimited text at once.
send_full_script=false

# Custom driver path to avoid copying your drivers
# driver_path=

Maven Goals

mvn migration:status

As the goal is straight forward, it provides the status of the migration whether the script has been executed or not. It provides a simple tabular information that contains id (timestamp provided in the filename), description (filename after the timestamp) and applied_at specifies when the migration has run for each file on the database.

mvn migration:up

A goal which will execute all pending migation scripts one by one based on the timestamp provided in the order.

mvn migration:down

A goal which will execute the undo section of the last migrated script.

mvn migration:pending

A goal which will execute all pending migation scripts one by one based on the timestamp provided in the order.

An alternate for MyBatis Migration is FlyWay and Liquidbase. You can download the fully functional working copy of the project from GitHub.

In case if you are wondering if we have any similar tool for MongoDB, then you can explore MongoBee which works similarly but only thing it lacks is reverting the changes as it’s not supported currently. To know more about it, please visit MongoBee.

Read More

Java String Concatenation

Image Source: https://www.educba.com/

Have you been told many times, don’t use + operator to concatenate Strings? We know that it is not good for performance. How do you really know whether is it true or not? Do you know what is happening behind the hood? Why don’t we go ahead and explore all about String concatenation?

In the initial versions of java around JDK 1.2 every body used + to concatenate two String literals. Strings are immutable, i.e., a String cannot be modified. Then what happens when we write the following code snippet.

• String message = "WE INNOVATE ";
• message = message + "DIGITAL";

In the above java code snippet for String concatenation, it looks like the String is modified but in reality it is not. Until JDK 1.4 the StringBuffer was used internally for concatenation and from JDK 1.5 StringBuilder is used to concatenate. After concatenation the resultant StringBuffer or StringBuilder is changed to String object.

You would have heard from java experts that, “don’t use + operator but use StringBuffer”. If + is going to use StringBuffer internally, what big difference it is going to make in String concatenation using + operator?

Look at the following examples. I have used both + and StringBuffer as two different cases.

• Case 01, I am just using + operator to concatenate.
• Case 02, I am changing the String to StringBuffer and then doing the concatenation. Then finally changing it back to String.

I have used a timer to record the time taken for an example of String concatenation.

public class StringConcatenateExample {
   private static final int LOOP_COUNT = 50000;
  
   public static void main(final String args[]) {
      long startTime, endTime;
    
      startTime = System.currentTimeMillis();
      String message = "*";
      for(int i=1; i<=LOOP_COUNT; i++) {
         message = message + "*";
      }
      endTime = System.currentTimeMillis() - startTime;
      System.out.println("Time taken to concatenate using + operator: " + endTime + " ms.");

      startTime = System.currentTimeMillis();
      StringBuilder sBuilder = new StringBuilder("*");
      for(int i=1; i<=LOOP_COUNT; i++) {
         sBuilder.append("*");
      }
      sBuilder.toString();
      endTime = System.currentTimeMillis() - startTime;
      System.out.println("Time taken to concatenate using StringBuilder: " + endTime + " ms.");
   }
  
}

Look at the output (if you run this java program the result numbers might slightly vary based on your hardware/software configuration). The difference between the two cases is extremely surprising.

You might argue, if + operator is using StringBuffer internally for concatenation, then why is this huge difference in time? Let me explain, when a + operator is used for concatenation, see how many steps are involved behind the scenes:

• A StringBuffer object is created.
• Message is copied to the newly created StringBuffer object.
• The “*” is appended to the StringBuffer (concatenation).
• The result is converted back to a String object.
• The message reference is made to point at that new String.
• The old String that message previously referenced is then made null.

It is now clear that there are serious performance issues that can result if you use + operator for concatenation and why is it so important to use StringBuffer or StringBuilder (from java 1.5) to concatenate Strings.

And on a side note, the StringBuffer is slower compared to StringBuilder because it’s a thread safe object, where all the methods are synchronised, so you need to take a decision wisely on usage based on your requirement.

Read More

Git Cherry Pick

Image Source: https://mattstauffer.com/

Some of my team members asked me how to merge only specific commits from a branch into the current branch. The reason you’d want to do this is to merge specific changes that you need immediately, leaving the other code changes you’re not interested.

First of all, use git log to see exactly which commit you want to pick or you can use the UI to identify the commit ID.

Let’s say you’ve written some code in the commit f69eb3 of the feature branch that is very important right at this moment. It may contain a bug fix or the code that other people need might need access to. You might want to have commit f69eb3 in the release branch, but not the other code you’ve written in the feature branch. Here the git cherry-pick comes very handy, in this case, f69eb3 is the cherry and you want to pick it.

Below are the step by step instructions to pick one commit from feature branch to release branch.

• git checkout release
• git cherry-pick f69eb3

That’s all, f69eb3 is now applied to the master branch and commited (as a new commit) in release branch. The cherry-pick behaves just like merge. If git can’t apply the changes then you will get merge conflicts. Git leaves you to resolve the conflicts manually and make the commit yourself.

In some cases picking one single commit is not enough. You may need, let’s say few consecutive commits. In this case, cherry-pick is not the right tool. Instead use rebase. From the previous example, you’d want commit 76f39a through b816a0 in release.

The process is to first create a new branch from feature at the last commit you want. Let’s say you want upto b816a0.

• git checkout -b mybranch b816a0

Next, you rebase the mybranch commit –onto master. The 76f39a^ indicates that you want to start from that specific commit.

• git rebase --onto master 76f39a

The result is that commits 76f39a through b816a0 are applied to master branch.

Please note, git commit ID is a hash of both its contents and its history. So, even if you have two commits that introduce the exact same change, if they point to different parent commits, they still have different IDs. After the cherry pick, the commit in the release branch will not reflect the same commit id as it will have new commit id.

Read More