Sunday, August 7, 2016

ACID (Part 4)

In the previous post, we discussed the Read Committed Isolation level. That level solved 1 type of read phenomena, Dirty Read, which we used to get in the Read Uncommited isolation level, but we still got Repeatable and phantom read with that level. In this post we talk about repeatable read isolation level. A slightly more expensive level of implementation but can kill the non-repeatable read phenomena.

The final state of our Like table look like this from the previous post.



Here is the reference to all ACID posts that I authored:

ACID (Part 1) - Atomicity and Consistency
ACID (Part 2) - Isolation (Read Uncommitted)
ACID (Part 3) - Isolation (Read Committed)
ACID (Part 4) - Isolation (Repeatable Read)


Repeatable Read

With a repeatable read isolation level, we not only read entries from committed transactions but we also read it from a previous timestamp. Usually it is the moment from we began our transaction, any other transactions that update entries after that moment will be overlooked by our transaction and it will attempt to retrieve an older "version" of the truth to make sure results are consistent. Lets jump to examples.


As we see, Eddard has already liked picture 2, he is attempting to fire another like to the same picture, we have atomicity and consistency that prevent him from doing so but lets see what happens. Eddard fires up a like on picture 2, and fraction of a second later Sansa loads up picture 2, this will retrieve the number of likes and the list of users who liked it. 

Eddard sends the like, first query executes successfully, incrementing the likes count.



Before Eddard second query executes, Sansa's Select kicks in to read picture 2 row, she is going to get 3 likes instead of 4. This is because we are operating under the repeatable read isolation level, which only reads committed transactions, and since Eddard still did not commit (or rollback) his transaction, Sansa is getting the current commited value. So we have avoided a dirty read phenomena.




Sansa issues another read to the Like table to get all the users who likes picture 2, she gets three rows, Jon, Reek and Eddard. Consistent with the number of likes she got.




Eddard transaction moves on and executes the second query which fails because of the constraint we have in place. Rolling back the entry for likes back to 3.







Eddard gives up, his transaction is finished and he failed to ruin our system consistent state, (we also chopped off his head). Meanwhile a new user comes in, Cersei and burns the database to the ground with wild fire. Not really, she likes picture 2, she is a brand new user who never liked picture 2 before so his transaction commits fine. 








Sansa's transaction is still running she is querying other tables, doing some stuff, updating the view count perhaps, and then finally, she comes back for a final read of picture 2 getting the likes count. Although the final committed value is 4, Sansa is going to get the original committed value when her transaction began which was 3. So we avoided getting the non-repeatable read phenamona with Repeatable read isolation level. It is slightly expensive since we have to keep history of versions of each committed values and go back searching for a previous value with a timestamp.





She issues a final read to the Like table to find out the list of users who liked picture 2, and surprise surprise, she got an extra record, hence Phantom read is still reproducible with repeatable read isolation level.






So we fix one problem with this isolation level but we introduce a cost of keeping history versions of previous committed values which we didn't have to with Read Committed level.

Next up, serializable isolation level.

-Hussein



Friday, July 29, 2016

4 Software patches that could make Fitbit blaze smarter

Got the fitbit blaze two months ago. I'm not a watch person but I really like it. However, there are few improvements that can be achieved by software only and make the watch smarter.

1. Exercise Termination

So when I start an exercise, for example biking, I go to Exercise -> Biking and then wait a for a moment for to establish a Bluetooth connection between the Blaze and the phone to pick up my GPS coordinate and other information from the smart phone. I then tap "Start" and bike. When I reach my destination, I manually tap "Finish" so it syncs my path, speed and all the data to the smart phone app.

I don't have a problem with manually stopping the exercise, but the problem is sometimes I start the biking exercise, and reach my destination and forget to tap "Finish", and if I don't do that the watch keeps running and gives false indication that I'm exercising when I'm actually not. One day it kept running for 8 hours (really 8 hours of continuous biking while I'm in the same location). And as I said I'm not a watch person so I'm not used to look at the watch often.

So there are many ways for fixing this. Fitbit could install a software patch for the watch that simply vibrate after a period of time (10 minutes or so) while in exercise mode. That will force me to look at the watch and manually stopping the exercise. There are more smart ways to solve this like checking the GPS coordinate, if I'm stationary for a while that means I'm not really biking and the exercise could terminate by it self. Although this might shorten the battery life as the watch will be forced to perform spatial calculation.


2. Offload Processing to Phone

Using the Blaze processor is expensive and can drain battery, so that should be minimized as much as  possible. While biking I noticed that the watch gets warmer and battery is quickly drained. The reason is the Blaze require the GPS coordinate to calculate the distance traveled, speed and other information. The GPS coordinate is acquired from the phone via the established Bluetooth connection. The Blaze then does all the processing on its processor, calculating the distance traveled, current speed. This could be optimized by offloading those extra processor cycles to the phone and periodically send the result to the blaze so it displays it.

Another option that could potentially optimize battery life is disable all those calculations while running the exercise. All the processing will be done on the phone and once the user terminate the exercise the result is sent to the watch and displayed. How may miles I biked (or walked), what was my average speed, maximum speed etc. This will optimize this even further for those (like me) who don't actually check their watch while biking and only check it once they reach their destination.

3. Power Saving Mode

There is a green beam light that detects whether the watch is on and I believe it also monitors the heart-beat as well. When I remove the watch, it still seem active and counts phantom steps and floors which wastes battery life. The blaze should switch to Power Saving mode or whatever you want to call it, which at that stage it shouldn't attempt to sync, count steps or floors. Charging this thing is the worst.


4. Alarm Sync

Although I don't wear it to bed, but it seems natural that I would like to sync my smart phone (IPhone or Andriod) to the Blaze. Having different set of alarms seem redundant.



-Hussein





Tuesday, July 26, 2016

ACID (Part 3)

So in the last post we discussed Isolation (the third letter from ACID). We managed to cover the 1st isolation level which was the Read Uncommitted. That level of isolation in fact offers no isolation at all (thus poor consistency), but yield excellent performance because we don't have to jump to previous "versions" of the record being read. The three phenomenas are reproducible with the read uncommitted. The next level of isolation we want to talk about is Read Committed.

Here is the reference to all ACID posts that I authored:

ACID (Part 1) - Atomicity and Consistency
ACID (Part 2) - Isolation (Read Uncommitted)
ACID (Part 3) - Isolation (Read Committed)
ACID (Part 4) - Isolation (Repeatable Read)

Read Committed

This isolation level offers isolation against uncommitted reads. A transaction will only read entries that has been committed by other transactions. So from this definition we know that dirty reads phenomena cannot appear in this isolation level. But again, we need a good example that illustrates this. Lets do our Instagram example again.

Brief recap, the Picture table has the LikesCount field which keeps track of how many likes a picture got. This is a performance tweak in order to avoid querying the Like table every time we want to get the likes count for a picture. We notice that picture 1 has 1 like only by Jon and picture 3 has two likes by Reek and Sansa. Picture 2 has 2 likes by Jon and Reek. The Picture_ID is the primary key in the Picture table, while both the Picture_ID and the User constitute the primary key for the Like table. There are many fields that we can add but for simplicity we are sticking with these now.






Jon has already liked picture 2, but he is insisting in liking it again, we have nice atomicity and consistency that prevent him from doing so but lets see what happens. Jon fires up a like on picture 2, and fraction of a second later Sansa loads up picture 2, this will retrieve the number of likes. 


Jon sends the like, first query executes successfully, incrementing the likes count.


Before Jon second query executes, Sansa's Select kicks in to read picture 2 row, she is going to get 2 likes instead of 3. This is because we are operating under the Read Committed isolation level, and since Jon still did not commit his transaction, Sansa is getting the current committed value. So we have avoided a dirty read phenomena. 



Sansa issues another read to the Like table to get all the users who likes picture 2, she gets two rows, Jon and Reek. Consistent with the number of likes she got.



Jon transaction moves on and executes the second query which fails because of the constraint we have in place. Rolling back the entry for likes back to 2.







Jon gives up, his transaction is finished and he failed to ruin our system consistent state. But meanwhile a new user comes in, Eddard, likes picture 2, he is a brand new user who never liked picture 2 before so his transaction commits fine.






Sansa's transaction is still running she is querying other tables, doing some stuff, updating the view count perhaps, and then finally, she comes back for a final read of picture 2 getting the likes count. She is getting a different result although she executed the same query twice in the same transaction. So we still get non-repeatable read phenamona with read committed isolation level.  




She issues a final read to the Like table to find out the list of users who liked picture 2, and surprise surprise, she got an extra record this time, hence Phantom read is also reproducible with read committed. 






So this isolation level did save us from a dirty read but still we getting phantoms and non-repeatable reads. The reason is that Read Committed reads committed transactions, and during the life time of the transaction a lot of other transactions can commit stuff that touches the data our transaction is about to read.

Next up, Repeatable Read

-Hussein







Friday, July 15, 2016

ACID (Part 2)

In the previous post, we discussed the two properties of ACID, Atomocity and Consistency.

Brief recap, the Picture table has the LikesCount field which keeps track of how many likes a picture got. This is a performance tweak in order to avoid querying the Like table every time we want to get the likes count for a picture. We notice that picture 1 has 1 like only by Jon and picture 3 has two likes by Reek and Sansa. Picture 2 has 1 like by Jon. The Picture_ID is the primary key in the Picture table, while both the Picture_ID and the User constitute the primary key for the Like table. There are many fields that we can add but for simplicity we are sticking with these now.

This is how our tables look like after the last post.


Here is the reference to all ACID posts that I authored:

ACID (Part 1) - Atomicity and Consistency
ACID (Part 2) - Isolation (Read Uncommitted)
ACID (Part 3) - Isolation (Read Committed)
ACID (Part 4) - Isolation (Repeatable Read)


Isolation

Obviously Jon is not the only user here, otherwise he would have ruled the Iron throne already, there are other users that concurrently liking pictures and updating the database. The question remain, how can we control what goes in first? should we go maximum isolation, stop everyone and use a first come first served approach so users don't step on each other's toes? or we should allow multiple users to update similar entries in parallel and deal with results? its our call really. First approach is slow but gives you consistent results, second approach much faster but might return incorrect results.

There are four types of isolation levels that we will try to cover.

Read Uncommitted

Here there is no isolation at all; Anything that is written, whether committed or not, is available to be read. Let us see how this performs in our Instagram scenario. 

Jon has already liked picture 2, but he is insisting in liking it again, we have nice atomicity and consistency that prevent him from doing so but lets see what happens. Jon fires up a like on picture 2, and fraction of a second later Sansa loads up picture 2, this will retrieve the number of likes. 


Jon sends the like, first query executes successfully, incrementing the likes count.


Before Jon second query executes, Sansa's Select kicks in to read picture 2, she is going to get (2 likes). This phenomena is called a dirty read, although Jon still didn't finish his transaction, we allowed Sansa to read a dirty uncommitted entry. 




Sansa issues another read to the Like table to get all the users who likes picture 2, she gets one row, which is Jon. Inconsistent with the number of likes she gets hmmm.





Jon transaction moves on and executes the second query which fails because of the constraint we have in place. Rolling back the entry for likes back to 1.








Sansa's transaction is still running she is querying other tables, doing some stuff, updating the view count perhaps, and then finally, she comes back for a final read of picture 2. She is getting a different result although she executed the same query twice in the same transaction, she got different results from each one. This phenomena is called Non-Repeatable read





While Sansa still executing her transaction, Reek likes picture 2, its his first like so the transaction commits fine.








Finally Sansa completes her transaction by reading the likes of picture 2 again, this time she gets an extra record! what a mess. She got different set of rows running the same query. This phenomena is called Phantoms read



With read uncommitted isolation level we got all three phenomenas. The question is how important isolation for us, can we tolerate these problems?


Next post Read Committed.

-Hussein


Wednesday, July 13, 2016

ACID (Part 1)

In the late 1970s,  Jim Gray, the computer scientist who went missing in 2007 and never found,  defined a set of properties for database transactions. These properties were Atomicity, Consistency, Isolation and Durability, thus ACID.

What are the ACID properties? How important are they? Can we trade these properties off for performance? Or should we architecture our software around those and make sure they all are met in all instances?

Here is the reference to all ACID posts that I authored:

ACID (Part 1) - Atomicity and Consistency
ACID (Part 2) - Isolation (Read Uncommitted)
ACID (Part 3) - Isolation (Read Committed)
ACID (Part 4) - Isolation (Repeatable Read)

This blog post and the following posts will attempt to answer these questions and clarify each property with hopefully a relatable example. I will try to be consistent and use one example to explain all four properties. I picked Instagram.

Instagram is a popular photo sharing service that is available on mobile devices. You can post a picture, and people who follow you can like or comment on those pictures.

Note: We are going to make a lot of assumption about Instagram database model which may not necessary be the actual implementation. However, we will try to make assumptions that yield highest performance.

Consider this database design, we have a Picture table, and Like table.




The picture table has the LikesCount field which keeps track of how many likes a picture got. This is a performance tweak in order to avoid querying the Like table every time we want to get the likes count for a picture. We notice that picture 1 has 1 like only by Jon and picture 3 has two likes by Reek and Sansa. Picture 2 has no likes. The Picture_ID is the primary key in the Picture table, while both the Picture_ID and the User constitute the primary key for the Like table. There are many fields that we can add but for simplicity we are sticking with these now.

Atomicity

Assume Jon, when he is not guarding the Wall, likes Picture 2. That is equivalent to one update query to the Picture table to increment the count of likes by 1 and one insert query to add a row in the Like table.


UPDATE PICTURE SET LIKESCOUNT = LIKESCOUNT +1 WHERE PICTURE_ID = 2;
INSERT INTO LIKE VALUES (2, 'Jon');

To achieve atomicity, those two queries should succeed together or they should fail together. They can't be one success and one fail or else we will end up in a inconsistent state where the likes rows do not match the count of the likes. Obviously its not a big deal if the like count did not match and here is where the trade off can happen. However, in some circumstances, like banking for instance, we can't tolerate such errors. The database solution should support atomic transactions where we can wrap those two queries in one transaction and then rollback in case one of them failed.

You might wonder, what could cause a query to fail? Network connectivity is one, hardware failure is another, but there are other constraints that we can put in place where the database can actually refuse to execute a query. Referential integrity and unique constraints for instance which brings us to the second property.





Consistency

Assume this, Jon really likes picture 2 he double clicked it again sending another like. Obviously this is wrong, but we have a constraint in the system to prevent that thank fully.


The first statement will go through updating the likes count..




The second query will fail because of unique constraint that we have put in place (Jon and 2) unique key already exists. This will cause the transaction to rollback and undo the first update as well, restoring the system to a consistent state. The atomicity kicked in here and saved the day in order to achieve consistency. If we didn't have an atomic transaction, we will end up with wrong results. Again not the end of the world.











So there is a lot of room for discussions here right, we can implement this logic at the client side to prevent Jon from sending the second like. But that mean an additional cost for reading that Like table and making sure Jon has already liked that. However, leaving a database constraints will lead to a lot of unnecessary traffic to the database that could be prevented.You guys can chip in for a better design here.

Another point I guess I should mention is a consistent state is different than a correct state. I would like to quote C.J. Date on this.

Consistent mean "satisfying all known integrity constraints" Observe, therefore, that consistent does not necessarily mean correct; a correct state must necessarily be consistent, but a consistent state might still be incorrect, in the sense that it does not accurately reflect the true state of affairs in the real world. "Consistent" might be defined as "correct as far as the system is concerned". - An introduction to Database system.
Consistency as you see is expensive to maintain, so some systems now especially distributed systems prefer a different consistency model called Eventual Consistency.  Which we will touch upon on another topic.

I want to handle Isolation in another post since it is a long topic.



Tuesday, July 5, 2016

Persistent Connections

English is not my first language, so I have to remind myself of definitions from time to time. So mind me if you see me define a phrase before I elaborate on it. To persist something is to keep it existing. So what does a persistent connection refer to? and why this is something software engineers should know about?

TCP is the de-facto protocol of the network communication. When we want to send data between node A and node B we establish a TCP connection, be it a simple visit to google.com or an Oracle database connection, under the hood its all TCP.  

Lots of things involve establishing the connection. Handshaking, acknowledgment, making sure the parties in the connection are in fact who they say they are etc. We don't need to discuss details of what exactly happens in this post, but one thing you should know, opening TCP connection is expensive. 

So imagine this scenario. You visit http://www.nationalgeographic.com website, this uses the HTTP protocol which underneath open a TCP connection to the national geographic server. So lets go back in time to 1996, when HTTP/1.0 was just released. This would happen when you visit that page:

Open TCP Connection national geographic website (this has many steps remember)
Read Index.html and send it back to the browser
Close TCP Connection
For each Image in the page
     Open TCP Connection
     Read Image from the server disk, send back to browser
     Close TCP Connection
Next image


Images are just an example of things we need to load, back then there were much more nastier resources. 

So now imagine the overhead of opening and closing the connections. The network congestion from all acknowledgments being sent back and forth and the wasted processing cycles both the server and the client have to endure. That is why persistent connections became popular, open a connection, and leave it open while we send everything we have, once we are done we can close it. Here is a modern visit of nationalgeographic.com in 2016, HTTP 1.1


Open TCP Connection national geographic website  
Read Index.html and send it back to the browser
For each Image in the page
     Read Image from the server disk, send back to browser
Next image
... Do more 
... Do more
Close TCP Connection


Now, I know this is very specific to browsers and web servers but the same story is true for database connections. In my years of experience as a programmer working with databases, I developed a habit, and am sure most of you did too, of opening a connection, sending a query and then closing a connection. That might be fine and barely noticeable if you have like 10 users working with your application. However, as you scale up, you will start noticing performance degradation.

Another thing you gain of persistent connections is PUSH events. This is how WhatsApp is able to freak you out by instantly delivering your wife's message "Where are you!" to your phone the moment she hit the send key on hers. WhatsApp do that by having a live open connection to their server from your mobile (not exactly TCP though, a much more efficient protocol called XMPP, we can touch upon that on some other post). 

Disadvantages?
We mentioned the advantages of persistent connection, but are there any disadvantages? Yes, there is no free lunch apparently.

When using persistent connection, you keep the connection alive on both the client and server, so you start eating up more memory with more connections opened. So you have to be smart about closing those idle connections.

Another problem came with persistent connections that is specifically for TCP. In a nut shell, now that we started to keep connections alive on the server, attackers came up with this idea, "Hey, what if we made the server run out of memory by establishing millions of connections and never replay back?" Thus DDOS attacks were born. Again we can touch more on this on another post. 

-Hussein

Monday, July 4, 2016

Process Isolation

You may have heard of this term in one of the conferences or tech talks. If you have worked with ArcGIS Server, you probably have seen the option to use High Process Isolation vs Low Process Isolation when configuring the GIS instances. So what does this mean and why does is it important to isolate processes?

So when we say "Isolate", we get a general idea of what that might be. To keep processes separate from each other. So you would think that one advantage of process isolation is to protect the process data from being accessed by other processes, and that is correct. But is that all?

Let's say you want to write a program to implement two functionalities (A and B).  Functionality A is low intensive (perhaps its just printing the result or updating the UI) and B is high intensive (querying the database, processing, rendering  etc..). So one approach is to write code to implement both A and B in one application.  Both Code A and B will be running in one process. 

Another approach is write the Code for A and the Code for B and use Process.Start() to execute code A in one process and code B in another process. Here is where the processes are isolated. 

In the first approach if both A and B runs in the same process, you risk B eating on your memory. If anything goes wrong in code B and it crashes, the entire process will crash taking code A down with it and users will lose your application entirely. However in the second approach, if B crashes, A will still be running and it can easily relaunch B, resurrect it back to life.

-Hussein

Saturday, April 30, 2016

ArcObjects SDK available for download !

After years finally users can freely download and work with the ArcObjects SDK for both .NET and Java! Previously this was only available to partners and customers who request the bundle DVD.

Here are the steps.

1. Go to Arcgis.com
2. Register a trial 60 days account
3. Sign in and on the right you can have the link.

Now that you have ArcObjects SDK, go build cool solutions.

Don't forget we have two Youtube series on IGeometry discussing how to get started with ArcObjects. One with VB.NET and another with C# , more than 30+ hours of content.

Enjoy

Extending ArcObjects (VB,NET)

https://www.youtube.com/watch?v=XrZs1rwmOwg&list=PLQnljOFTspQXqYsWJG8o-eJpGlvzww9lE

ArcGIS Add-incs  (C#)

https://www.youtube.com/watch?v=nn4mtxdi19A&list=PLQnljOFTspQVMRyBp7UOt8pU7wygBmEja&src_vid=XrZs1rwmOwg&feature=iv&annotation_id=annotation_401451585


Tuesday, April 26, 2016

See the Great American Eclipse of August 21, 2017

My colleague Mike Zeiler is fascinated with Maps. What is astonishing is how he is able to author to serve his hobby. Solar Eclipses.

He travels around the world to see every eclipse, he is among the few who actually does that. Next year the eclipse will be in United States - Oregon, so he doesn't have to travel too far this time.

This is his website, go check out the cool maps he built and pay attention to the count down timer to the next eclipse.

http://www.greatamericaneclipse.com/




Enjoy

Monday, April 25, 2016

ArcGIS for Server Architecture Explained



A youtube episode (part of docker series) where we discuss the ArcGIS for Server architecture. We talk about web servers (web adaptors) GIS Servers, config stores and how each component play a role in the ArcGIS server architecture.
Enjoy