Travis invalidate aws CloudFront cache

How to make aws CloudFront cache invalidation in travis CI.
NB: In previous articles You can always find how to set up deployment in travis ci to aws.
So, let say we need to deploy static files to aws s3 (example) let’s add “after_deploy” section to .travis.yml:

lang: ruby
#...
deploy:
  provider: s3
#...
after_deploy:
  - ./.travis/reset_cloudfront_cache.sh $SITE_KEY $SITE_KEY_SECRET

Note: that $SITE_KEY and $SITE_KEY_SECRET must be travis encrypted values (details). Also, this user (which key and secret we are going to use) must have CloudFront permissions, so do not forget to add him to the appropriate group in aws/IAM.

For the .travis.yml config that’s it, we just need to run reset_cloudfront_cache script from the after_deploy section (so it will be started after deploy is done).

Let’s take a look at reset_cloudfront_cache.sh:

###############################################################################
###  Resets CloudFront cache with boto/cfadmin utility
###  Run: ./this_script  
###############################################################################

#
# Travis specific part - run this script only for production
#

# If this is fork - just exit
if [[ -n "${TRAVIS_PULL_REQUEST}" && "${TRAVIS_PULL_REQUEST}" != "false"  ]]; then
  echo -e '\n============== deploy will not be started (from the fork) ==============\n'
  exit 0
fi

if [[ $TRAVIS_BRANCH == 'master' ]]; then
    echo -e "\nThis is master/production branch - let's reset the CloudFront cache\n"
else
    echo -e "\nReset of CloudFront cache will not be started for non-production branch - exit.\n"
    exit 0
fi

#
# Install boto
#
echo -e "\nInstalling boto...\n"
git clone git://github.com/boto/boto.git
cd boto
sudo python setup.py install
cd ../
rm -rf boto

#
# Set up credentials for boto
#
echo -e "\nSet up boto credentials...\n"
cat > ~/.boto < 
# XNXNXXNNNNXNXN - distribution configured for the Web - in aws amazon cloudfront distributions.
#
echo -e "\nCloudFront Invalidating...\n"
cfadmin invalidate XNXNXXNNNNXNXN /yoursite.com
cfadmin invalidate XNXNXXNNNNXNXN /.yoursite.com
cfadmin invalidate XNXNXXNNNNXNXN /index.html /static/main.css /static/main.js
echo -e "\nInvalidating is in progress...\n"
echo -e "\nYou can check the status on the 'Invalidations' tab here https://console.aws.amazon.com/cloudfront/home?region=your_region#distribution-settings:XNXNXXNNNNXNXN\n"

#
# Clean up
#
echo -e "\nRemove boto config file\n"
rm ~/.boto

The main idea of this script is to use boto/cfadmin utility to reset specific paths in the selected distribution (this is library based on CloudFront rest API).
So we need to install boto first, then create ~/.boto config file (with aws credentials) and just to reset\invalidate the CoudFront cache with the following command:

cfadmin invalidate <distribution> <path>

That will start the Invalidation process that may last up to 10-30 minutes.

Social Share Toolbar

Travis branch specific variable

How to set different values for variables in travis travis.yml based on the current branch?
Let’s take a look at the part of the travis config:

lang: ruby
env:
  global:
  - GET_ENV_VAR=./.travis/get_env_specific_value.sh
  #
  # Keys:
  #
  # SITE_DEV_KEY
  - secure: "..."
  # SITE_DEV_KEY_SECRET
  - secure: "..."
  # SITE_PROD_KEY
  - secure: "..."
  # SITE_PROD_KEY_SECRET
  - secure: "..."
  #
  # Branch specific variables:
  #
  - ENV=$($GET_ENV_VAR develop master)
  - SITE_KEY=$($GET_ENV_VAR $SITE_DEV_KEY $SITE_PROD_KEY)
  - SITE_KEY_SECRET=$($GET_ENV_VAR $SITE_DEV_KEY_SECRET $SITE_PROD_KEY_SECRET)
  - AWS_BUCKET=$($GET_ENV_VAR dev-bucket prod-bucket)
  - AWS_REGION=$($GET_ENV_VAR us-west-2 us-west-1)

The magic happens in this line

GET_ENV_VAR=./.travis/get_env_specific_value.sh

To be more specific the very magic happens in this script “./.travis/get_env_specific_value.sh”, let’s take a look at the source code of this bash script:

#!/bin/bash
 
# If this is fork - just exit
if [[ -n "${TRAVIS_PULL_REQUEST}" && "${TRAVIS_PULL_REQUEST}" != "false"  ]]; then
  echo -e '\n============== deploy will not be started (from the fork) ==============\n'
  exit 0
fi
 
# Choose necessary argument according to the current branch.
if [[ $TRAVIS_BRANCH == 'develop' ]]; then
    echo $1
elif [[ $TRAVIS_BRANCH == 'master' ]]; then
    echo $2
else
    exit 0
fi

The idea of this script is just to return 1-st or 2-nd argument depends on the current branch name.

Social Share Toolbar

Travis ci deploy static files to amazon s3

How to deploy static files to aws s3 bucket?
Here is the example of travis.yml config:

lang: ruby
env:
  global:
  # env variables should be defined here:
  - AWS_BUCKET=...
before_script:
  # workaround to fix rake not found
  - gem install rake
deploy:
  provider: s3
  access_key_id: $SITE_KEY
  secret_access_key: $SITE_KEY_SECRET
  bucket: $AWS_BUCKET
  region: $AWS_REGION
  skip_cleanup: true
  acl: public_read
  on:
    all_branches: true

To workaround problem with Rakefile not found you need to add this Rakefile:

task :default do
  puts "default rake task"
end
Social Share Toolbar

Talk from UX people conference

My talk from uxpeople.by:
video http://vimeo.com/85995792 (Russian)

Social Share Toolbar

Validation in angular.js ng-repeat

It seems to be a kind of pretty obvious task to make validation for fields generated with angular ng-repeat directive, but actually I think it is not so obvious as I thought. Let say we have a dynamic “table” of “fields”, both fields are numbers and left one should be less then right one. BTW let say that every line of this table is a range, so the next “low” value should be as previous “high” + 1, e.g.:
[1] [2]
[3] [10]
[11] [20]

Let’s create simple form for this:

<div ng-app='formValidationApp'>
<form ng-controller="ValidationCtrl" name='rangesForm' novalidate>
    <div class='line' ng-repeat='line in ranges'>
        low: <input type='text' ng-pattern='/^\d+$/' ng-model='line.low' />
        high: <input type='text' ng-pattern='/^\d+$/' ng-model='line.up' />
        <a href ng-if='!$first' ng-click='removeRange($index)' >Delete</a>
    </div>
    <a href ng-click='addRange()'>Add Range</a>
    <input type='submit' ng-disabled='rangesForm.$invalid'>
</form>
</div>

And here is the controller:

formValidationApp = angular.module('formValidationApp', []);
formValidationApp.controller('ValidationCtrl', ['$scope', function ($scope){
    var scope_ = $scope;
    $scope.ranges = [{low: 1}, {}, {}];
    $scope.removeRange = function (index){
        scope_.ranges.splice(index, 1);
    };
    $scope.addRange = function(){
        scope_.ranges.push({});
    };
}]);

And little bit CSS:

.line {
    clear: both;
}
input.ng-invalid {
    border-color: red;
    color: red;
    background-color: pink;
}

The working code you can find in this fiddle http://jsfiddle.net/82PX4/

Ok, as you may see I’ve added basic angular validation via “ng-pattern”. Also I want to add “validation error text”, as far as you may know interpolation doesn’t work in ng-repeat, I mean you can not set input’s name to something like “low_$index”. As an example http://stackoverflow.com/questions/12044277/how-to-validate-inputs-dynamically-created-using-ng-repeat-ng-show-angular

Ok, but we have “ng-form” for this case, cool let’s use it. I added ng-form to the DIV with the repeater, and added validation error, so here is the updated html:

<div ng-app='formValidationApp'>
<form ng-controller="ValidationCtrl" name='rangesForm' novalidate>
    <div class='line' ng-repeat='line in ranges' ng-form='lineForm'>
        low: <input type='text' ng-pattern='/^\d+$/' ng-model='line.low' />
        high: <input type='text' ng-pattern='/^\d+$/' ng-model='line.up' />
        <a href ng-if='!$first' ng-click='removeRange($index)' >Delete</a>
        <div class='error' ng-show='lineForm.$error.pattern'>
            Must be a number.
        </div>
    </div>
    <a href ng-click='addRange()'>Add Range</a>
    <input type='submit' ng-disabled='rangesForm.$invalid' />
</form>
</div>

And here is the updated jsfiddle – http://jsfiddle.net/82PX4/1/

Ok that’s fine, and now let’s try to add custom validation for inputs in angular ng-repeat directive.
I won’t create angular directive, because initially the main goal of directive is “reusability”. Let say that this “range” validation is completely custom thing and this is the only usage of such kind of logic. The creation of directive will simplify the implementation of this task, but for me it’s kind of “hack” in this case, the logic is very custom and should be in controller\service\etc but not in directive.

The most efficient way (in terms of angular) to validate our “model” (I mean $scope.ranges) is to watch $scope.ranges and to validate it (the logic will operate only with data, and will watch only data), but afterward, when for example first range will be invalid, it is needed to set validity for this element and this is the problem.

It is impossible to access defined (let say the second) form (lineForm) because interpolation for form’s name doesn’t work, so it is impossible to make something like this:

rangesForm.lineForm[2].low.$setValidity('ranges', false);

or like this

rangesForm.lineForm_2.low.$setValidity('ranges', false);

That’s a pity, I hope I just didn’t find the right solution to access element by index.

The only solution that I found is to use “ng-change” for tracking changes (instead of watching the data model) and to pass “this” to the ng-change handler (so we can access the scope of current ng-form).

So, lets modify the code. Here is the ng-repeat section:

<div class='line' ng-repeat='line in ranges' ng-form='lineForm'>
        low: <input type='text' 
                    name='low'
                    ng-pattern='/^\d+$/' 
                    ng-change="lowChanged(this, $index)" ng-model='line.low' />
        up: <input type='text' 
                    name='up'
                    ng-pattern='/^\d+$/'
                    ng-change="upChanged(this, $index)" 
                    ng-model='line.up' />
        <a href ng-if='!$first' ng-click='removeRange($index)' >Delete</a>
        <div class='error' ng-show='lineForm.$error.pattern'>
            Must be a number.
        </div>
        <div class='error' ng-show='lineForm.$error.range'>
            Low must be less the Up.
        </div>
    </div>

And here is updated js:

formValidationApp = angular.module('formValidationApp', []);
formValidationApp.controller('ValidationCtrl', ['$scope', function ($scope){
    var scope_ = $scope;
    $scope.ranges = [{low: 1}, {}, {}];
    $scope.removeRange = function (index){
        scope_.ranges.splice(index, 1);
    };
    $scope.addRange = function(){
        scope_.ranges.push({});
    };
    $scope.lowChanged = function(scope, index) {
        // check ranges
        setValidity(
            scope.ranges[index].low,
            scope.ranges[index].up,
            scope.lineForm.low);
 
    };
    $scope.upChanged = function(scope, index) {
        // check ranges
        setValidity(
            scope.ranges[index].low,
            scope.ranges[index].up,
            scope.lineForm.up);
    }
 
    function setValidity(low, up, element) {
        if(low && up && !!+low && !!+up) {
            element.$setValidity('range', 
                parseInt(low) < parseInt(up));
        }
    }
}]);

Here is the working fiddle http://jsfiddle.net/82PX4/2/

Let’s little bit improve UX and add “auto filling” (I mean to set next Up value to previous Low value +1, and vice versa).
So, here is the complete solution – http://jsfiddle.net/82PX4/3/

I’m not sure that this solution is good enough, so if you know better one, please send me a mail (gmail [the doggy] Mikita Manko [dot] com)

Social Share Toolbar

MongoDB performance bottlenecks, optimization Strategies for MongoDB

I will try to describe here all potential performance bottlenecks and possible solutions and tips for performance optimization, but first of all – You should to ensure that MongoDB was the right choice for your project. You should clearly understand that MongoDB is completely “Nonrelational Database” (I mean no joins). And MongoDB is Document orientated database (not graph oriented). This is completely Important to be sure that you made the right choice of database.

0. Map-Reduce
Before MongoDB 2.4 update (main point here is update to V8 engine) MongoDB have been using SpiderMonkey as a javascript engine, and the problem was that it’s single threaded (that was pretty awkward when Map-Reduce has been working only on 1 core from e.g. 48 ones). So after the 2.4 update performance was raised up, but there are too many pitfalls and you’d better to read this out http://docs.mongodb.org/manual/core/map-reduce/ and this one http://docs.mongodb.org/manual/core/aggregation-pipeline/.

N.B. Bear in mind that the performance of Map-Reduce is depends upon “the state of data”, I mean, that the difference between Map-Reduce on the data “as-is” and on the sorted data is too huge (on sorted data Map-Reduce will be something like 10-100x faster then without sorting). So, to raise up the performance of Map-Reduce you need:

  • Find out the key, that you will use for Map-Reduce job (usually it’s the same as the emit key) and ENSURE that you have added indexes for this key (you can try to run your query filter)
  • Add input sort for key for the Map-Reduce job(emit key)

Also, please take a look at this doc http://blog.mongodb.org/post/62900213496/qaing-new-code-with-mms-map-reduce-vs-aggregation

1. Sharding
It is hella cool to have out of the box sharding, but apart from the sharding you have also one of the performance pitfalls.
Shard keys should satisfy the following:

  • “distributable” – the worst case of the shard key is auto-incremented value (this will entail the “hot shard” behavior, when all writes will be balanced to the single shard – here is the bottle neck). Ideal shard key should be as much “randomness” as possible.
  • Ideal shard key should be the primary field used for your queries.
  • An easily divisible shard key makes it easy for MongoDB to distribute content among the shards. Shard keys that have a limited number of possible values can result in chunks that are “unsplittable.”
  • unique fields in your collection should be part of the shard key

Here is the doc about shard key

2. Balancing
You should bear in mind that moving chunks from shard to another shard is a very expensive operation (adding of new shards may significantly slow down the performance).
As an helpful option – you could stop the balancer during the “prime time”.

3. Disk Input Output operations
You should understand that in most cases the hardware bottleneck will be HDD (not CPU or RAM), especially if you have several shards.
So, during the growth of data, the number of I/O operations will rapidly increase. Also keep monitoring free disk space. So fast disks are more important in case if you are using sharding.

4. Locks
MongoDB uses a readers-writer lock that allows concurrent reads access to a database but gives exclusive access to a single write operation.
When a read lock exists, many read operations may use this lock. However, when a write lock exists, a single write operation holds the lock exclusively, and no other read or write operations may share the lock.
Locks are “writer greedy,” which means writes have preference over reads. When both a read and write are waiting for a lock, MongoDB grants the lock to the write.

And the very sad point – MongoDB implements locks on a per-database basis for most read and write operations (before 2.2 update was the global lock – one per instance for al databases).
This is very valuable point, and if you have too many write requests here will be the bottleneck with the solution (m.b. it’s really to create hack with several databases, but better forget about this).

In case if your application have too many write operations it make sense to think about migration to something like Cassandra (In Cassandra, a write is atomic at the row-level, meaning inserting or updating columns for a given row key will be treated as one write operation).
Please take a look at concurrency docs to ensure that you understanding mongo concurrency.

5. Fast Writes
Use Capped Collections for Fast Writes
Capped Collections are circular, fixed-size collections that keep documents well-ordered, even without the use of an index. This means that capped collections can receive very high-speed writes and sequential reads.

These collections are particularly useful for keeping log files but are not limited to that purpose. Use capped collections where appropriate.

6. Fast Reads
Use Natural Order for Fast Reads. To return documents in the order they exist on disk, return sorted operations using the $natural operator. On a capped collection, this also returns the documents in the order in which they were written.
Natural order does not use indexes but can be fast for operations when you want to select the first or last items on disk.

7. Query Performance
Read out about query performance, especially please pay attention to Indexes and Compound Indexes.

8. Remove Expired Data
It seems to be a good practice to enable the TTL (time to live) in your collections, add expireAfterSeconds value and use Expire Data from Collections by Setting TTL technique. This approach will allow you to get rid of “unnecessary data”.

9. The size of Database
As far as you might understand MongoDB will store e.g. this document

{	UserFirstAndLastName: "Mikita Manko",
	LinkToUsersFacebookPage: "https://www.facebook.com/mikita.manko"
}

“as-is”. I mean that names of these fields “UserFirstAndLastName” and “LinkToUsersFacebookPage” will reduce free space.
Buy the using “name shorting” technique you can minimize the usage of memory (you can get rig of something like 30-40% of unnecessary data):

{	FL: "Mikita Manko",
	BFL: "https://www.facebook.com/mikita.manko"
}

Obviously that it will cause the creation of “mapper” in your code (You should map shortened unreadable names from database to long ones to allow to use readable fields in your code)

A. Application Design
Take a look at these notes and bear them in mind during the designing of your architecture and solutions.

B. Profiling and Investigations
You should be familiar with such features as:

C. Updates
The most obvious point is to be on the cutting edge of technologies and Investigate and Install last updates.

P.S.
As i mentioned before – Use MongoDB not just for fun, but if your project is applicable for Document Oriented Database, that is the most important point.

Social Share Toolbar

My slides from UxPeople.by

My slides from http://uxpeople.by/http://www.slideshare.net/mikitamanko/mikita-manko-at-u-xpeopleby

Social Share Toolbar

Little Snake on JavaScript (30 lines of code)

Inspired by Excel in 30 lines I challenged myself to create something fun in 30 lines of code. I choose the game Snake. After almost two hours of “home made” hackathone I created this Fiddle

That’s not pretty good code, but challenge was resolved without usage of any library:)

Some details:

  • I removed code for generation of markup
  • There is no any array, the algorithm is “DOM addicted” – I mark with class “s” snake’s cells and with “f” class cells with Food (So it’s obvious how to detect collision with snake or with food).
  • Every new snake’s cell will be marked with “data-n” attribute (incrementing integer value)
  • little hack how to remove unnecessary tale cell – find all divs with class “s” (all snake’s calls) then find div with minimal “data-n” value (it’s the oldest cell – the tale of the snake) and make this cell usual (remove the “s” class – remove the tale)
(function(width, height, length, current, dx, dy, x, y, hasFood, newEl){     
 
document.body.onkeydown = function(e){
    dx = (e.keyCode - 38) % 2, dy = (e.keyCode - 39) % 2;
};
 
var timer = setInterval(function () {
    x = (x + dx) < 0 ? width - 1 : (x + dx) % width; 
    y = (y + dy) < 0 ? height - 1 : (y + dy) % height;
    newEl = document.getElementsByClassName(y + '_' + x)[0]
    if(newEl.className.indexOf('s') > 0) {
    	clearInterval(timer), alert('Game Over! Score: ' + length)
    };
    if(newEl.className.indexOf('f') > 0) {
    	newEl.className = newEl.className.replace(' f', ''), length++, hasFood = false;
    }
    newEl.className += ' s', newEl.setAttribute('data-n', current++);
 
    for(var i = 0, min = Infinity, item, items = document.getElementsByClassName('s'), len = items.length; i < len && len > length; i++)
    	if(+items[i].getAttribute('data-n') < min)
    		min = +items[i].getAttribute('data-n'), item = items[i];
 
    if(!!item) item.className = item.className.replace(' s', '');
 
    for(var fItem, fX, fY; !hasFood; fX = Math.round(Math.random() * 10 % width), fY = Math.round(Math.random() * 10 % height))
    	if(!!fX && !!fY && document.getElementsByClassName(fY + '_' + fX)[0].className.indexOf('s') < 0)
    		hasFood = true, document.getElementsByClassName(fY + '_' + fX)[0].className += ' f';
}, 1000);
 
})(10, 10, 5, 1, 1, 0, 0, 0, false, null);

P.S. code is not good enough even for hackathone but seems that it works fine. Also it’s only 30 lines of code.

Social Share Toolbar

Chrome remote debugging on ios 6 (iphone 5)

Let’s dig into chrome remote debugging on ios 6 (iphone 5)

1. First of all we need to configure fiddler as a proxy
open fiddler->tools->fiddler options-> connections: set port to 8888, check “allow remote computers to connect” (google how to set up fiddler proxy for details)

2. configure iphone:
open command line on your desktop and type “ipconfig” to figure out your local ip
open settigns on iphone->wi-fi->current active network settings: scroll down, set proxy to manual and fill in proxy ip (ip from previous point and port 8888)

3. check that iphone is using your desktop as a proxy
just open any page in chrome on iphone and check fiddler out

4. install nodejs (click install here http://nodejs.org/)
check installation with the following commands:

node -v

I got something like – “v0.10.17”
then check that npm has been also installed

npm -v

I got something like 0 “1.3.8”

5. configure local server for files
e.g. we can add to hosts file “127.0.0.1 mydomain.com”
and set up new IIS site for this domain, so we got this page “http://mydomain.com/mypage.html” mapped on local env

6. configure local page – we need to add this script to our local page

<script src="http://mydomain:8080/target/target-script-min.js#anonymous"></script>

7. run weinre – run from console

weinre --boundHost mydomain.com

should respond something like “2013-08-26T13:31:58.109Z weinre: starting server at http://mydomain.com:8080”

8. open debugger
open this link http://mydomain.com:8080/client/#anonymous

9. open page on a mobile device – open page with weinre script – http://mydomain.com/mypage.html

10. check out desktop chrome with opened http://mydomain.com:8080/client/#anonymous – this debugger should be attached to the page opened on iphone

11. Thats all

P.S.
some helpful links:
http://people.apache.org/~pmuellr/weinre/docs/latest/Installing.html
http://people.apache.org/~pmuellr/weinre/docs/latest/Running.html

Social Share Toolbar

Mongoose aggregate with $group by nested field

Let’s say we have a collection in mongodb with document’s structure similar to this:

// schema on nodejs via mongoose:
var mongoose = require('mongoose')
    , Schema = mongoose.Schema
    , ObjectId = Schema.ObjectId;
 
var flatSchema = new Schema({
    _id: ObjectId,
    rentType: Number,
    price: Number,
    thumbnail: String,
    address: {
        country: String,
        city: String,
        street: String,
        district: Number
        // ... more fields
    }
    // ... some more fields
}, { collection: 'flats'});
 
module.exports = mongoose.model('Flat', flatSchema);

and we need to get some items (that matched to some request) and then group all documents by “address.district” (nested field) and after this we also need to collect some stats – e.g. min and max price and flats count for each district.

Let’s do it with usage of Mongoose:

    // e.g. let's define request
    var rules = [{'address.city': '1'}, {price: {$gte: 200}}];
 
    // and here are the grouping request:
    Flat.aggregate([
        { $match: {$and: rules } },
        {
            $project: {
                _id: 0, // let's remove bson id's from request's result
                price: 1, // we need this field
                district: '$address.district' // and let's turn the nested field into usual field (usual renaming)
            }
        },
        {
            $group: {
                _id: '$district', // grouping key - group by field district
                minPrice: { $min: '$price'}, // we need some stats for each group (for each district)
                maxPrice: { $max: '$price'},
                flatsCount: { $sum: 1 }
            }
        }
    ], {}, callback);

First of all – it’s not possible to group by nested field, e.g. “address.district”, thats why we workarounded this by field’s renaming in $project.

Here is the list of interesting reading:
http://docs.mongodb.org/manual/reference/aggregation/group/
http://docs.mongodb.org/manual/reference/aggregation/project/
http://mongoosejs.com/docs/api.html#model_Model.aggregate

That was simple example of how to group something via mongoose on nodejs without usage of direct access to mongo’s db.collection.

Social Share Toolbar