Notes on eventual consistency

Consistency guarantee definitions

From most to least strict:

  1. Strong Consistency – Reads see all previous writes.
  2. Eventual Consistency – Reads see subset of previous writes.
  3. Consistent Prefix – Reads see an initial sequence of writes.
  4. Bounded Staleness – Reads see all “old” writes.
  5. Monotonic Reads – Reads see an increasing subset of writes.
  6. Read My Writes – Reads see all writes performed by the reader.

Chapter from “Designing Data-Intensive applications” book:

Reading Your Own Writes

Many applications let the user submit some data and then view what they have submitted. This might be a record in a customer database, or a comment on a discussion thread, or something else of that sort. When new data is submitted, it must be sent to the leader, but when the user views the data, it can be read from a follower. This is especially appropriate if data is frequently viewed but only occasionally written.

With asynchronous replication, there is a problem: if the user views the data shortly after making a write, the new data may not yet have reached the replica. To the user, it looks as though the data they submitted was lost, so they will be understandably unhappy.

A user makes a write, followed by a read from a stale replica. To prevent this anomaly, we need read-after-write consistency.
A user makes a write, followed by a read from a stale replica. To prevent this anomaly, we need read-after-write consistency.

In this situation, we need read-after-write consistency, also known as read-your-writes consistency [24]. This is a guarantee that if the user reloads the page, they will always see any updates they submitted themselves. It makes no promises about other users: other users’ updates may not be visible until some later time. However, it reassures the user that their own input has been saved correctly.

How can we implement read-after-write consistency in a system with leader-based replication? There are various possible techniques. To mention a few:

  • When reading something that the user may have modified, read it from the leader; otherwise, read it from a follower. This requires that you have some way of knowing whether something might have been modified, without actually querying it. For example, user profile information on a social network is normally only editable by the owner of the profile, not by anybody else. Thus, a simple rule is: always read the user’s own profile from the leader, and any other users’ profiles from a follower.
  • If most things in the application are potentially editable by the user, that approach won’t be effective, as most things would have to be read from the leader (negating the benefit of read scaling). In that case, other criteria may be used to decide whether to read from the leader. For example, you could track the time of the last update and, for one minute after the last update, make all reads from the leader. You could also monitor the replication lag on followers and prevent queries on any follower that is more than one minute behind the leader.
  • The client can remember the timestamp of its most recent write—then the system can ensure that the replica serving any reads for that user reflects updates at least until that timestamp. If a replica is not sufficiently up to date, either the read can be handled by another replica or the query can wait until the replica has caught up. The timestamp could be a logical timestamp (something that indicates ordering of writes, such as the log sequence number) or the actual system clock (in which case clock synchronization becomes critical; see “Unreliable Clocks”).
  • If your replicas are distributed across multiple datacenters (for geographical proximity to users or for availability), there is additional complexity. Any request that needs to be served by the leader must be routed to the datacenter that contains the leader.

Another complication arises when the same user is accessing your service from multiple devices, for example a desktop web browser and a mobile app. In this case you may want to provide cross-device read-after-write consistency: if the user enters some information on one device and then views it on another device, they should see the information they just entered.

In this case, there are some additional issues to consider:

  • Approaches that require remembering the timestamp of the user’s last update become more difficult, because the code running on one device doesn’t know what updates have happened on the other device. This metadata will need to be centralized.
  • If your replicas are distributed across different datacenters, there is no guarantee that connections from different devices will be routed to the same datacenter. (For example, if the user’s desktop computer uses the home broadband connection and their mobile device uses the cellular data network, the devices’ network routes may be completely different.) If your approach requires reading from the leader, you may first need to route requests from all of a user’s devices to the same datacenter.

Monotonic Reads

Our second example of an anomaly that can occur when reading from asynchronous followers is that it’s possible for a user to see things moving backward in time.

This can happen if a user makes several reads from different replicas. For example, image shows user 2345 making the same query twice, first to a follower with little lag, then to a follower with greater lag. (This scenario is quite likely if the user refreshes a web page, and each request is routed to a random server.) The first query returns a comment that was recently added by user 1234, but the second query doesn’t return anything because the lagging follower has not yet picked up that write. In effect, the second query is observing the system at an earlier point in time than the first query. This wouldn’t be so bad if the first query hadn’t returned anything, because user 2345 probably wouldn’t know that user 1234 had recently added a comment. However, it’s very confusing for user 2345 if they first see user 1234’s comment appear, and then see it disappear again.

A user first reads from a fresh replica, then from a stale replica. Time appears to go backward. To prevent this anomaly, we need monotonic reads.
A user first reads from a fresh replica, then from a stale replica. Time appears to go backward. To prevent this anomaly, we need monotonic reads.

Monotonic reads is a guarantee that this kind of anomaly does not happen. It’s a lesser guarantee than strong consistency, but a stronger guarantee than eventual consistency. When you read data, you may see an old value; monotonic reads only means that if one user makes several reads in sequence, they will not see time go backward—i.e., they will not read older data after having previously read newer data.

One way of achieving monotonic reads is to make sure that each user always makes their reads from the same replica (different users can read from different replicas). For example, the replica can be chosen based on a hash of the user ID, rather than randomly. However, if that replica fails, the user’s queries will need to be rerouted to another replica.

Consistent Prefix Reads

Our third example of replication lag anomalies concerns violation of causality. Imagine the following short dialog between Mr. Poons and Mrs. Cake:

Mr. Poons
How far into the future can you see, Mrs. Cake?
Mrs. Cake
About ten seconds usually, Mr. Poons.

There is a causal dependency between those two sentences: Mrs. Cake heard Mr. Poons’s question and answered it.

Now, imagine a third person is listening to this conversation through followers. The things said by Mrs. Cake go through a follower with little lag, but the things said by Mr. Poons have a longer replication lag. This observer would hear the following:

Mrs. Cake
About ten seconds usually, Mr. Poons.
Mr. Poons
How far into the future can you see, Mrs. Cake?

To the observer it looks as though Mrs. Cake is answering the question before Mr. Poons has even asked it. Such psychic powers are impressive, but very confusing.

If some partitions are replicated slower than others, an observer may see the answer before they see the question.
If some partitions are replicated slower than others, an observer may see the answer before they see the question.

Preventing this kind of anomaly requires another type of guarantee: consistent prefix reads. This guarantee says that if a sequence of writes happens in a certain order, then anyone reading those writes will see them appear in the same order.

This is a particular problem in partitioned (sharded) databases, which we will discuss in Chapter 6. If the database always applies writes in the same order, reads always see a consistent prefix, so this anomaly cannot happen. However, in many distributed databases, different partitions operate independently, so there is no global ordering of writes: when a user reads from the database, they may see some parts of the database in an older state and some in a newer state.

One solution is to make sure that any writes that are causally related to each other are written to the same partition—but in some applications that cannot be done efficiently. There are also algorithms that explicitly keep track of causal dependencies, a topic that we will return to in “The “happens-before” relationship and concurrency”.

Solutions for Replication Lag

When working with an eventually consistent system, it is worth thinking about how the application behaves if the replication lag increases to several minutes or even hours. If the answer is “no problem,” that’s great. However, if the result is a bad experience for users, it’s important to design the system to provide a stronger guarantee, such as read-after-write. Pretending that replication is synchronous when in fact it is asynchronous is a recipe for problems down the line.

As discussed earlier, there are ways in which an application can provide a stronger guarantee than the underlying database—for example, by performing certain kinds of reads on the leader. However, dealing with these issues in application code is complex and easy to get wrong.

It would be better if application developers didn’t have to worry about subtle replication issues and could just trust their databases to “do the right thing.” This is why transactions exist: they are a way for a database to provide stronger guarantees so that the application can be simpler.

Single-node transactions have existed for a long time. However, in the move to distributed (replicated and partitioned) databases, many systems have abandoned them, claiming that transactions are too expensive in terms of performance and availability, and asserting that eventual consistency is inevitable in a scalable system. There is some truth in that statement, but it is overly simplistic, and we will develop a more nuanced view over the course of the rest of this book.

Notes from JDD 2017

Documentation, it is alive!

Tools to keep documentation up to date without putting too much effort:

Readme templates/suggestions:

Consumer Driven Contracts (CDC)

Performance testing

  • Run performance tests at least once and save the results
  • Run stability tests at least once
  • Don’t forget to run them on environment similar to production
  • This isn’t as much of time consuming as many think


Package by feature (OOD approach)

  • Create mid-size building blocks using packages and package-scope
  • Those packages are your modules, this is your architecture
  • public = published interface
  • Remove public from all your IDE templates
  • Test modules as black boxes via interactions with it and other modules (no internal state verification)
  • Write acceptance tests as unit tests (bloody fast and easy)
  • Write integration tests to check wiring, I/O, configuration, positive flow (again)
  • It’s actually simpler than not doing it
  • Make building blocks that people can understand and use easily


A note from “Microservices AntiPatterns and Pitfalls”

Comparison of different approaches on how to build reporting service over distributed system.


Database pull model
Database pull model: reporting system has direct access to service database. Data context is not bounded to specific service.
HTTP pull model
HTTP pull model: reporting system has access only to API of the service. Data context is encapsulated, but speed of report generation is degrading quickly with growth of services amount.
Batch pull model
Batch pull model: data context is not bounded, reports are generated over long period of times.
Event-based push model
Event-based push model: solves bounded context problem and performance issues, although adds complexity to services – each of them has to know how and which data to push
Model comparison
Model comparison

Types of ML problems


A classifier uses a set of instances for which the correct category membership is known.

Questions: spam or ham, positive or negative.

Training Data: ex. tweets which are correctly classified as positive or negative.


Forecasting of continuous value.

Questions: what will be the price of this stock on a given date, what will be a sales of this product in a future week

Training Data: historical datapoints.


Helps to identify the groups in raw dataset (groups of users in social network).

We’re telling how many groups there should be and algorithm makes grouping objects by different attributes into this number of groups.

Recommendations (Collaborative Filtering)

Determine what user may like based on past behavior.


Notes from JDD 2016

Topic: Flying to cloud
Main idea: How to move an app to the cloud, how easily this can be done with spring-cloud, spring wraps lots of libraries from netflix to make it integrated with its own ecosystem + adds some default behavior

Good book: Migrating to cloud native application architecture (
Possible problems: Global locks, leadership election

Spring cloud benefits:
Circuit breaker (histryx) works OOTB
@FeignClient and @EnableFeignClients allows to group replicated clients and add load balancing (with Ribbon annotation), also OOTB

media-20161010-1 media-20161010-2 media-20161010


Topic: Fast feedback
Main idea: Lots of upcoming problems could be solved by achieving the fast feedback in many different areas. Author describes many techniques he uses on his projects

When throwing exception: add some context on different stages, then rethrow – this practice eliminates time spent on the issue analysis
mvn – T – allows parallel build execution (has many options):

mvn -T 4 clean install # Builds with 4 threads
mvn -T 1C clean install # 1 thread per cpu core
mvn -T 1.5C clean install # 1.5 thread per cpu core

Static analysis:

  • use blacklist (filter) instead of whitelist of rules, to stay up to date with new rules suggested by PMD, FindBugs etc.
  • ben-manes plugin scans your project and shows if there are new libraries to use
  • mutation testing:
  • NonDex – analyze wrong assumptions about Java api
  • CodeNarc – static code analyzer for Groovy

Topic: Security
Main idea: Author gives some advices on how to keep quality of the security on the project

Letsencrypt – gives free SSL certificates (yet you should pay for hosting of the certificate)
OWASP ASVS – security standards for developer
ZAP Zed attack proxy – helps to find security vulnerabilities
SAMM Overview – maturity model

Topic: Non-blocking microservices
Main idea: How one team moved their code and infrastructure to non-blocking IO

Someone had calculated that 1 thread ~1mb, which costs 8$ per year to maintain
First they’ve looked into JAX-RX 2.0 which supports async responses.
Non-blocking servlet servlet will require non-blocking filters, which have to be written customly
So they moved to pure HTTP.
CompletableFuture (Java 8) allows to chain futures (like promises in JS).
Migration requires functional decomposition, to allow easy chaining.
Recommended Netty + async http client.
Move to NIO rather in case of low CPU loads, to benefit

Topic: Metrics gathering
Main idea: How one team gathered metrics

They’re using io.dropwizard.metrics:
– metrics-core
– metrics-spring – spring integration
– metrics-jvm – collect gc metrics
Works with @Timed annotation

Graphite and graphana – ui and storage for metrics
Read More

Power of grep

How to highlight results in ls output:

ls|grep -E --color "sh$|$"

This will highlight all files with *.sh extension in current directory.

Where -E means that we’re matching entries with regex,
|$ means that we don’t want to skip entries that do not match our regex (passing pipe further and closing it with dollar sign).