23 Jan

Types of ML problems

Classification

A classifier uses a set of instances for which the correct category membership is known.

Questions: spam or ham, positive or negative.

Training Data: ex. tweets which are correctly classified as positive or negative.

Regression 

Forecasting of continuous value.

Questions: what will be the price of this stock on a given date, what will be a sales of this product in a future week

Training Data: historical datapoints.

Clustering

Helps to identify the groups in raw dataset (groups of users in social network).

We’re telling how many groups there should be and algorithm makes grouping objects by different attributes into this number of groups.

Recommendations (Collaborative Filtering)

Determine what user may like based on past behavior.

 

13 Nov

Notes from JDD 2016

Topic: Flying to cloud
Main idea: How to move an app to the cloud, how easily this can be done with spring-cloud, spring wraps lots of libraries from netflix to make it integrated with its own ecosystem + adds some default behavior

Good book: Migrating to cloud native application architecture (http://12factor.net)
Possible problems: Global locks, leadership election

Spring cloud benefits:
Circuit breaker (histryx) works OOTB
@FeignClient and @EnableFeignClients allows to group replicated clients and add load balancing (with Ribbon annotation), also OOTB

media-20161010-1 media-20161010-2 media-20161010

 


Topic: Fast feedback
Main idea: Lots of upcoming problems could be solved by achieving the fast feedback in many different areas. Author describes many techniques he uses on his projects

When throwing exception: add some context on different stages, then rethrow – this practice eliminates time spent on the issue analysis
mvn – T – allows parallel build execution (has many options):

mvn -T 4 clean install # Builds with 4 threads
mvn -T 1C clean install # 1 thread per cpu core
mvn -T 1.5C clean install # 1.5 thread per cpu core

Static analysis:

  • use blacklist (filter) instead of whitelist of rules, to stay up to date with new rules suggested by PMD, FindBugs etc.
  • ben-manes plugin scans your project and shows if there are new libraries to use
  • mutation testing: pitest.org
  • NonDex – analyze wrong assumptions about Java api
  • CodeNarc – static code analyzer for Groovy

Topic: Security
Main idea: Author gives some advices on how to keep quality of the security on the project

Letsencrypt – gives free SSL certificates (yet you should pay for hosting of the certificate)
OWASP ASVS – security standards for developer
ZAP Zed attack proxy – helps to find security vulnerabilities
SAMM Overview – maturity model


Topic: Non-blocking microservices
Main idea: How one team moved their code and infrastructure to non-blocking IO

Someone had calculated that 1 thread ~1mb, which costs 8$ per year to maintain
First they’ve looked into JAX-RX 2.0 which supports async responses.
BUT:
Non-blocking servlet servlet will require non-blocking filters, which have to be written customly
So they moved to pure HTTP.
CompletableFuture (Java 8) allows to chain futures (like promises in JS).
Migration requires functional decomposition, to allow easy chaining.
Recommended Netty + async http client.
Move to NIO rather in case of low CPU loads, to benefit


Topic: Metrics gathering
Main idea: How one team gathered metrics

They’re using io.dropwizard.metrics:
– metrics-core
– metrics-spring – spring integration
– metrics-jvm – collect gc metrics
Works with @Timed annotation

Graphite and graphana – ui and storage for metrics
Read More

17 Apr

Power of grep

How to highlight results in ls output:

ls|grep -E --color "sh$|$"

This will highlight all files with *.sh extension in current directory.

Where -E means that we’re matching entries with regex,
|$ means that we don’t want to skip entries that do not match our regex (passing pipe further and closing it with dollar sign).

22 Oct

Apace CXF for REST application

I started to configure my first RESTful app using Apache CXF library and bumped into issue with JAX-RS binding. No matter how I tried to map my classes, I’ve still been getting this error in response instead of JSON data:

No message body writer has been found for response class ArrayList.

The solution was to add dependency on JSON mapping provider since Apache CXF does not include it. It can be XStream or Jackson, I used the last one.

Step 1.

Add dependency for org.codehaus.jackson:jackson-jaxrs to your project

Step 2.

Add JSON provider to Spring context configuration:

<beans xmlns="http://www.springframework.org/schema/beans"...>
...
    <jaxrs:server id="serviceId">
        <jaxrs:serviceBeans>
            <ref bean="myService"/>
        </jaxrs:serviceBeans>
        <jaxrs:providers>
            <bean class="org.codehaus.jackson.jaxrs.JacksonJsonProvider"/>
        </jaxrs:providers>
    </jaxrs:server>
...
</beans>

 

12 Sep

Vim game

To learn Vim with fun I started to search for some interactive guide with some practical step-by-step tasks.

I found a game that helps to remember the main Vim shortcuts:

git clone git://github.com/kikuchiyo/vim_game.git

Although I needed to spend some time to find out how to compile Ruby app with all dependencies.
Here is the recipe that worked for me:

First of all you need have installed MySQL server and Ruby itself:

sudo apt-get install mysql-server
sudo apt-get install ruby

Then install bundler (dependency tool for Ruby):

sudo apt-get install ruby-bundler

You’ll also need Qt 4 (couldn’t completely compile app with Qt 5 but I guess it should be installed as well):

sudo apt-get install qt4-default

Install dependencies:

sudo apt-get install libqt5webkit5-dev
sudo apt-get install qtquick1-5-dev qtlocation5-dev qtsensors5-dev qtdeclarative5-dev
sudo apt-get install libsqlite3-dev libmysql-ruby libmysqlclient-dev

Then try to set up the application itself:

bundle install
bundle exec rake db:drop db:create db:schema:load db:migrate
bundle exec rails s

Enjoy!