Comparison of different approaches on how to build reporting service over distributed system.
A note from “Microservices AntiPatterns and Pitfalls”
Types of ML problems
Classification
A classifier uses a set of instances for which the correct category membership is known.
Questions: spam or ham, positive or negative.
Training Data: ex. tweets which are correctly classified as positive or negative.
Regression
Forecasting of continuous value.
Questions: what will be the price of this stock on a given date, what will be a sales of this product in a future week
Training Data: historical datapoints.
Clustering
Helps to identify the groups in raw dataset (groups of users in social network).
We’re telling how many groups there should be and algorithm makes grouping objects by different attributes into this number of groups.
Recommendations (Collaborative Filtering)
Determine what user may like based on past behavior.
Notes from JDD 2016
Topic: Flying to cloud
Main idea: How to move an app to the cloud, how easily this can be done with spring-cloud, spring wraps lots of libraries from netflix to make it integrated with its own ecosystem + adds some default behavior
Good book: Migrating to cloud native application architecture (http://12factor.net)
Possible problems: Global locks, leadership election
Spring cloud benefits:
Circuit breaker (histryx) works OOTB
@FeignClient and @EnableFeignClients allows to group replicated clients and add load balancing (with Ribbon annotation), also OOTB
Topic: Fast feedback
Main idea: Lots of upcoming problems could be solved by achieving the fast feedback in many different areas. Author describes many techniques he uses on his projects
When throwing exception: add some context on different stages, then rethrow – this practice eliminates time spent on the issue analysis
mvn – T – allows parallel build execution (has many options):
mvn -T 4 clean install # Builds with 4 threads mvn -T 1C clean install # 1 thread per cpu core mvn -T 1.5C clean install # 1.5 thread per cpu core
Static analysis:
- use blacklist (filter) instead of whitelist of rules, to stay up to date with new rules suggested by PMD, FindBugs etc.
- ben-manes plugin scans your project and shows if there are new libraries to use
- mutation testing: pitest.org
- NonDex – analyze wrong assumptions about Java api
- CodeNarc – static code analyzer for Groovy
Topic: Security
Main idea: Author gives some advices on how to keep quality of the security on the project
Letsencrypt – gives free SSL certificates (yet you should pay for hosting of the certificate)
OWASP ASVS – security standards for developer
ZAP Zed attack proxy – helps to find security vulnerabilities
SAMM Overview – maturity model
Topic: Non-blocking microservices
Main idea: How one team moved their code and infrastructure to non-blocking IO
Someone had calculated that 1 thread ~1mb, which costs 8$ per year to maintain
First they’ve looked into JAX-RX 2.0 which supports async responses.
BUT:
Non-blocking servlet servlet will require non-blocking filters, which have to be written customly
So they moved to pure HTTP.
CompletableFuture (Java 8) allows to chain futures (like promises in JS).
Migration requires functional decomposition, to allow easy chaining.
Recommended Netty + async http client.
Move to NIO rather in case of low CPU loads, to benefit
Topic: Metrics gathering
Main idea: How one team gathered metrics
They’re using io.dropwizard.metrics:
– metrics-core
– metrics-spring – spring integration
– metrics-jvm – collect gc metrics
Works with @Timed annotation
Graphite and graphana – ui and storage for metrics
Read More
Show current Mercurial branch in shell (like in Git Bash)
hgBranch() {
if [ -d .hg ]; then
printf "$1" "$(hg branch)"
fi
}
export PS1='${USER}@${HOSTNAME}:\w$(hgBranch " [\033[1;31m(%s)\033[0m]")$ '
Power of grep
How to highlight results in ls output:
ls|grep -E --color "sh$|$"
This will highlight all files with *.sh extension in current directory.
Where -E means that we’re matching entries with regex,
|$ means that we don’t want to skip entries that do not match our regex (passing pipe further and closing it with dollar sign).
How to move several commits between branches in Mercurial (from stackoverflow)
Enable mq extension
[extensions] mq=
Do next things
hg qimport -r 1:3 # convert revisions to patches hg qpop -a # remove all them from history hg branch new # start a new branch hg qpush -a # push them all back into history hg qfin -a # finalize the patches
Enabling CamelCase navigation in IntelliJ IDEA
To enable Eclipse-like CamelCase navigation in IntelliJ IDEA follow these steps:
Step 1
Go to Settings > Editor > Smart Keys and check Use “CamelHumps” words.
Step 2
Go to Settings > Editor and check Honor “CamelHumps” words settings when selecting on double click
Apace CXF for REST application
I started to configure my first RESTful app using Apache CXF library and bumped into issue with JAX-RS binding. No matter how I tried to map my classes, I’ve still been getting this error in response instead of JSON data:
No message body writer has been found for response class ArrayList.
The solution was to add dependency on JSON mapping provider since Apache CXF does not include it. It can be XStream or Jackson, I used the last one.
Step 1.
Add dependency for org.codehaus.jackson:jackson-jaxrs to your project
Step 2.
Add JSON provider to Spring context configuration:
<beans xmlns="http://www.springframework.org/schema/beans"...>
...
<jaxrs:server id="serviceId">
<jaxrs:serviceBeans>
<ref bean="myService"/>
</jaxrs:serviceBeans>
<jaxrs:providers>
<bean class="org.codehaus.jackson.jaxrs.JacksonJsonProvider"/>
</jaxrs:providers>
</jaxrs:server>
...
</beans>
Vim game
To learn Vim with fun I started to search for some interactive guide with some practical step-by-step tasks.
I found a game that helps to remember the main Vim shortcuts:
git clone git://github.com/kikuchiyo/vim_game.git
Although I needed to spend some time to find out how to compile Ruby app with all dependencies.
Here is the recipe that worked for me:
First of all you need have installed MySQL server and Ruby itself:
sudo apt-get install mysql-server sudo apt-get install ruby
Then install bundler (dependency tool for Ruby):
sudo apt-get install ruby-bundler
You’ll also need Qt 4 (couldn’t completely compile app with Qt 5 but I guess it should be installed as well):
sudo apt-get install qt4-default
Install dependencies:
sudo apt-get install libqt5webkit5-dev sudo apt-get install qtquick1-5-dev qtlocation5-dev qtsensors5-dev qtdeclarative5-dev sudo apt-get install libsqlite3-dev libmysql-ruby libmysqlclient-dev
Then try to set up the application itself:
bundle install bundle exec rake db:drop db:create db:schema:load db:migrate bundle exec rails s
Enjoy!