tag:blogger.com,1999:blog-39176044095470437132024-03-13T03:59:31.634+02:00Obscured by CloudsCloud Computing. About Innovation, Technologies, DesignAdrian Copiehttp://www.blogger.com/profile/02438930394274756287noreply@blogger.comBlogger12125tag:blogger.com,1999:blog-3917604409547043713.post-59502300795820217112015-08-22T12:16:00.002+03:002015-08-22T12:16:24.866+03:00Saving Google Forms data in Google Spreadsheet <div dir="ltr" style="text-align: left;" trbidi="on">
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; text-align: justify;">
<span style="background-color: transparent; color: black; font-family: 'Times New Roman'; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">These days I dealt with a </span><a href="https://www.google.com/forms/about/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: 'Times New Roman'; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Google Forms</span></a><span style="background-color: transparent; color: black; font-family: 'Times New Roman'; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> application which did many things, breaking a little the standard use cases, and I considered useful for those who want to squeeze more from its functionality, to share some points from the quest of developing it. From the very beginning, Google Forms is a great application if you want to design a form in just a few clicks and collect the submitted data. Very useful when you want to implement a survey or, if you are a teacher, you could easily prepare an examination grid and collect the answers from your students.</span></div>
<b id="docs-internal-guid-abf9c8a4-54af-ee73-29fb-dafb07c5669e" style="font-weight: normal;"><br /></b>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; text-align: justify;">
<span style="background-color: transparent; color: black; font-family: 'Times New Roman'; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Google Forms helps to design a user interface, offering a collection of predefined form controls, allowing some validations over the fields, but it is a little rigid in respect with the look and user experience, I mean here the positioning in the page and customization. In fact, this is not its purpose, and they can be ignored. Once the design of the form being done, the application is ready to collect the submitted data, in a spreadsheet, without writing a line of code. </span></div>
<b style="font-weight: normal;"><br /></b>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; text-align: justify;">
<span style="background-color: transparent; color: black; font-family: 'Times New Roman'; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">If you want to do more with your form, </span><a href="https://developers.google.com/google-apps/products" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: 'Times New Roman'; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Apps Script</span></a><span style="background-color: transparent; color: black; font-family: 'Times New Roman'; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> programming language is your ally. Every application from Google Apps suite has its own programming API, exposing different objects and methods. The application framework supports stand-alone methods and event handlers. You can write modular code spreaded in many distinct files (having .gs extension) but at the end it is the same as they are in a single big file. To debug the code, Google offers an integrated debugger that can execute step-by-step, and a Logger object which helps log some useful data to an execution console. Besides this, there is an Execution Transcript process which logs every interpreted line of code. </span></div>
<b style="font-weight: normal;"><br /></b>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; text-align: justify;">
<span style="background-color: transparent; color: black; font-family: 'Times New Roman'; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Suppose you collect all the data from a form in a spreadsheet, but besides this, you want to have one or more columns in the spreadsheet based on some processed data after the form submission. This is not one of the usual use cases, so a little coding is necessary. First of all, two applications are involved: Google Forms and Google Scripts. The flow could be:</span></div>
<b style="font-weight: normal;"><br /></b>
<ol style="margin-bottom: 0pt; margin-top: 0pt;">
<li dir="ltr" style="background-color: transparent; color: black; font-family: 'Times New Roman'; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; text-align: justify;">
<span style="background-color: transparent; color: black; font-family: 'Times New Roman'; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Collect the data in Google Forms</span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: 'Times New Roman'; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; text-align: justify;">
<span style="background-color: transparent; color: black; font-family: 'Times New Roman'; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Submit the form</span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: 'Times New Roman'; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; text-align: justify;">
<span style="background-color: transparent; color: black; font-family: 'Times New Roman'; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">In the form submit handler of Google Forms, open the background spreadsheet and write the processed data</span></div>
</li>
</ol>
<b style="font-weight: normal;"><br /></b>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; text-align: justify;">
<span style="background-color: transparent; color: black; font-family: 'Times New Roman'; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Or the flow could be:</span></div>
<b style="font-weight: normal;"><br /></b>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; text-align: justify;">
<span style="background-color: transparent; color: black; font-family: 'Times New Roman'; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> Collect the data in Google Forms</span></div>
<ol start="4" style="margin-bottom: 0pt; margin-top: 0pt;">
<li dir="ltr" style="background-color: transparent; color: black; font-family: 'Times New Roman'; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; text-align: justify;">
<span style="background-color: transparent; color: black; font-family: 'Times New Roman'; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Submit the form</span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: 'Times New Roman'; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; text-align: justify;">
<span style="background-color: transparent; color: black; font-family: 'Times New Roman'; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">In the form submit handler of Google Spreadsheet, get the collected data from the form and write the processed data</span></div>
</li>
</ol>
<b style="font-weight: normal;"><br /></b>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; text-align: justify;">
<span style="background-color: transparent; color: black; font-family: 'Times New Roman'; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Both flows look pretty the same but let's see what happens. </span></div>
<b style="font-weight: normal;"><br /></b>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; text-align: justify;">
<span style="background-color: transparent; color: black; font-family: 'Times New Roman'; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">In the first case, the form submission handler is attached to the Google Form app:</span></div>
<b style="font-weight: normal;"><br /></b>
<div dir="ltr" style="margin-left: 0pt;">
<table style="border-collapse: collapse; border: none; width: 624px;"><colgroup><col width="*"></col></colgroup><tbody>
<tr style="height: 0px;"><td style="background-color: #f3f3f3; border-bottom: solid #000000 1px; border-left: solid #000000 1px; border-right: solid #000000 1px; border-top: solid #000000 1px; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-size: 12px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><span style="font-family: Courier New, Courier, monospace;">function onFormSubmit(e){</span></span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-size: 12px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><span style="font-family: Courier New, Courier, monospace;"> var processedData = "Data processed from some of the form fields";</span></span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-size: 12px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><span style="font-family: Courier New, Courier, monospace;"> </span></span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-size: 12px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><span style="font-family: Courier New, Courier, monospace;"> var sheet = SpreadsheetApp.openById("1gIqbtpsttN5Z8EKuHBrA1YI0eqPh6AGQo6tYtsFm7YY");</span></span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-size: 12px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><span style="font-family: Courier New, Courier, monospace;"> </span></span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-size: 12px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><span style="font-family: Courier New, Courier, monospace;"> var activeRange = e.range;</span></span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-size: 12px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><span style="font-family: Courier New, Courier, monospace;"> </span></span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-size: 12px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><span style="font-family: Courier New, Courier, monospace;"> var rangeToWrite = sheet.getRange(activeRange.getRow(), activeRange.getLastColumn()+1);</span></span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-size: 12px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><span style="font-family: Courier New, Courier, monospace;"> rangeToWrite.setValue(processedData); </span></span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-size: 12px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><span style="font-family: Courier New, Courier, monospace;">}</span></span></div>
</td></tr>
</tbody></table>
</div>
<b style="font-weight: normal;"><br /></b>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; text-align: justify;">
<span style="background-color: transparent; color: black; font-family: 'Times New Roman'; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Running this code will fall into:</span></div>
<b style="font-weight: normal;"><br /></b>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; text-align: justify;">
<span style="background-color: transparent; color: black; font-family: 'Times New Roman'; font-size: 14.666666666666666px; font-style: italic; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Execution failed: You do not have permission to call openById (line XX, file "Code") [0.002 seconds total runtime]</span><span style="background-color: transparent; color: black; font-family: 'Times New Roman'; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> </span></div>
<b style="font-weight: normal;"><br /></b>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; text-align: justify;">
<span style="background-color: transparent; color: black; font-family: 'Times New Roman'; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">because Google Forms app tried to open the SpreadsheetApp, and I didn’t find any solution to make it work. </span></div>
<b style="font-weight: normal;"><br /></b>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; text-align: justify;">
<span style="background-color: transparent; color: black; font-family: 'Times New Roman'; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">A workaround for this situation is to place the code in the onFormSubmit handler of the Google Spreadsheet.</span></div>
<b style="font-weight: normal;"><br /></b>
<div dir="ltr" style="margin-left: 0pt;">
<table style="border-collapse: collapse; border: none; width: 624px;"><colgroup><col width="*"></col></colgroup><tbody>
<tr style="height: 0px;"><td style="background-color: #f3f3f3; border-bottom: solid #000000 1px; border-left: solid #000000 1px; border-right: solid #000000 1px; border-top: solid #000000 1px; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-size: 12px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><span style="font-family: Courier New, Courier, monospace;">function onFormSubmit(e){</span></span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-size: 12px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><span style="font-family: Courier New, Courier, monospace;"> var processedData = "Data processed from some of the form fields";</span></span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-size: 12px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><span style="font-family: Courier New, Courier, monospace;"> </span></span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-size: 12px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><span style="font-family: Courier New, Courier, monospace;"> var sheet = SpreadsheetApp.getActive().getSheetByName("Form Responses 1"); </span></span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-size: 12px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><span style="font-family: Courier New, Courier, monospace;"> var activeRange = e.range;</span></span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-size: 12px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><span style="font-family: Courier New, Courier, monospace;"> </span></span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-size: 12px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><span style="font-family: Courier New, Courier, monospace;"> var rangeToWrite=sheet.getRange(activeRange.getRow(), activeRange.getLastColumn()+1);</span></span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-size: 12px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><span style="font-family: Courier New, Courier, monospace;"> rangeToWrite.setValue(processedData); </span></span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-size: 12px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><span style="font-family: Courier New, Courier, monospace;">}</span></span></div>
</td></tr>
</tbody></table>
</div>
<b style="font-weight: normal;"><br /></b>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; text-align: justify;">
<span style="background-color: transparent; color: black; font-family: 'Times New Roman'; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">In this case the value fields is received through the Event object </span><span style="background-color: transparent; color: black; font-family: 'Source Code Pro'; font-size: 12px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">e</span><span style="background-color: transparent; color: black; font-family: 'Times New Roman'; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> and no need to open the Forms App, to fall in the same error, so the write operation works fine.</span></div>
<b style="font-weight: normal;"><br /></b>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; text-align: justify;">
<span style="background-color: transparent; color: black; font-family: 'Times New Roman'; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Other useful things to consider is about sending emails from inside Google Spreadsheets. Usually MailApp is used for this, but you cannot change the “From” field of the email. The From address will be always the email address of the entity that created the trigger for form submission. If GmailApp is used instead of MailApp, the From field can be changed to an alias of the entity that created the trigger, but not to a complete different email address.</span></div>
<b style="font-weight: normal;"><br /></b>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; text-align: justify;">
<span style="background-color: transparent; color: black; font-family: 'Times New Roman'; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> </span></div>
<br /></div>
Adrian Copiehttp://www.blogger.com/profile/02438930394274756287noreply@blogger.com0tag:blogger.com,1999:blog-3917604409547043713.post-74147443955963568382013-01-05T22:58:00.000+02:002013-01-05T22:58:00.945+02:00<div dir="ltr" style="text-align: left;" trbidi="on">
<span style="font-size: large;"><b>Setting riak to store data in an EBS volume in Amazon EC2</b></span><br />
<br />
These days I've performed some tests in a riak cluster configuration, involving about a million write operations from many virtual machines hosted in EC2. Due to the database replication and the number and size of the records to be written, the default storage space coming with an Ubuntu Server 12.04LT instance was insufficient, so the solution was to add extra storage space through Elastic Storage Blocks (EBS).<br />
<br />
The procedure is quite simple and will be described below in order to save the time for those who need to do this storage extension.<br />
<br />
First of all a new volume must be created from AWS console and attach it to the desired instance.<br />
<br />
See how the volume is attached in Ubuntu by isuing the command:<br />
<br />
<b id="internal-source-marker_0.8770390609279275" style="font-weight: normal;"><span style="font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><span style="font-family: Courier New, Courier, monospace;">df</span></span><br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><br /></span></b><div>
<b style="font-weight: normal;"><span style="font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><span style="font-family: Times, Times New Roman, serif;">The result of the command is:</span></span></b></div>
<div>
<b style="font-weight: normal;"><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><br /></span></b></div>
<div>
<b style="font-weight: normal;"><span style="font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><span style="font-family: Courier New, Courier, monospace;">Filesystem 1K-blocks Used Available Use% Mounted on</span></span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">/dev/xvda1 8256952 793960 7043564 11% /</span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">udev 838328 8 838320 1% /dev</span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">tmpfs 338520 172 338348 1% /run</span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">none 5120 0 5120 0% /run/lock</span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">none 846300 0 846300 0% /run/shm</span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">/dev/xvdb 153899044 192068 145889352 1% /mnt</span><br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span>The new volume is </b><b style="font-weight: normal;"><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">/dev/xvdb </span><span style="font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><span style="font-family: Times, Times New Roman, serif;">and prior to use it it must be formatted appropriately. For doing this:</span></span></b><b><br /><br /><span style="font-size: 15px; font-weight: normal; vertical-align: baseline; white-space: pre-wrap;"><span style="font-family: Times, Times New Roman, serif;">Unmount the</span><span style="font-family: Courier New, Courier, monospace;"> /dev/xvdb</span><span style="font-family: Times, Times New Roman, serif;"> volume:</span></span><br /><span style="font-family: Arial; font-size: 15px; font-weight: normal; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-family: 'Courier New'; font-size: 15px; font-weight: normal; vertical-align: baseline; white-space: pre-wrap;">sudo umount /dev/xvdb</span><br /><span style="font-family: Arial; font-size: 15px; font-weight: normal; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><span style="font-family: Times, Times New Roman, serif;"><span style="font-weight: normal;">Format the partition using </span>ext4<span style="font-weight: normal;"> </span></span></span><br /><span style="font-family: 'Courier New'; font-size: 15px; font-weight: normal; vertical-align: baseline; white-space: pre-wrap;"><br /></span></b></div>
<div>
<b style="font-weight: normal;"><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">sudo mkfs.ext4 -j /dev/xvdb</span><br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><span style="font-family: Times, Times New Roman, serif;">Mount back the volume:</span></span></b></div>
<div>
<b style="font-weight: normal;"><span style="font-family: Times, Times New Roman, serif;"><span style="font-size: 15px; white-space: pre-wrap;"><br /></span></span><span style="font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><span style="font-family: Courier New, Courier, monospace;">sudo mount /dev/xvdb</span></span><br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><span style="font-family: Times, Times New Roman, serif;">Find out how was the EBS volume mounted:</span></span><br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">df</span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><br /></span></b></div>
<div>
<b style="font-weight: normal;"><span style="font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><span style="font-family: Times, Times New Roman, serif;">The result of the command is:</span></span></b></div>
<div>
<b style="font-weight: normal;"><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><br /></span></b></div>
<div>
<b style="font-weight: normal;"><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">Filesystem 1K-blocks Used Available Use% Mounted on</span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">/dev/xvda1 8256952 860640 6976884 11% /</span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">udev 838328 8 838320 1% /dev</span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">tmpfs 338520 172 338348 1% /run</span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">none 5120 0 5120 0% /run/lock</span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">none 846300 0 846300 0% /run/shm</span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">/dev/xvdb 153899044 191936 145889484 1% /mnt</span><br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><span style="font-family: Times, Times New Roman, serif;">Modify riak configuration file</span></span><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> </span></b><b style="font-weight: normal;"><span style="font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><span style="font-family: Courier New, Courier, monospace;">/etc/riak/app.config</span></span></b><b style="font-weight: normal;"><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">:</span></b></div>
<div>
<b style="font-weight: normal;"><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><span style="font-family: Times, Times New Roman, serif;">Find the</span></span><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> </span><span style="font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><span style="font-family: Courier New, Courier, monospace;">riak_core</span></span><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> </span><span style="font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><span style="font-family: Times, Times New Roman, serif;">setting and modify</span></span><br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><span style="font-family: Courier New, Courier, monospace;">{platform_data_dir, "/var/lib/riak"}</span></span><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> </span><span style="font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><span style="font-family: Times, Times New Roman, serif;">to:</span></span><span style="font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><span style="font-family: Courier New, Courier, monospace;"> </span></span></b></div>
<div>
<b style="font-weight: normal;"><span style="font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><span style="font-family: Courier New, Courier, monospace;">{platform_data_dir, "/mnt/var/lib/riak"}</span></span><br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><span style="font-family: Times, Times New Roman, serif;">and </span><span style="font-family: Courier New, Courier, monospace;">bitcask </span><span style="font-family: Times, Times New Roman, serif;">setting</span></span><br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><span style="font-family: Courier New, Courier, monospace;">{data_root, "/var/lib/riak/bitcask"} </span></span><span style="font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><span style="font-family: Times, Times New Roman, serif;">to:</span></span><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> </span></b></div>
<div>
<b style="font-weight: normal;"><span style="font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><span style="font-family: Courier New, Courier, monospace;">{data_root, "/mnt/var/lib/riak/bitcask"}</span></span><br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><span style="font-family: Times, Times New Roman, serif;">On the new formatted volume mounted under </span><span style="font-family: Courier New, Courier, monospace;">/mnt</span><span style="font-family: Times, Times New Roman, serif;"> prepare the directories that will hold the date on the new volume:</span></span></b></div>
<div>
<b style="font-weight: normal;"><span style="font-family: Times, Times New Roman, serif;"><span style="font-size: 15px; white-space: pre-wrap;"><br /></span></span><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">/mnt/var/lib/riak/bitcask</span></b><b id="internal-source-marker_0.8770390609279275" style="font-weight: normal;"><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">/mnt/var/lib/riak/kv_node</span></b> <br />
<br />
and make them writable.<br />
<br />
Then, start the riak service by issuing the command:<br />
<br />
<span style="font-family: Courier New, Courier, monospace;">sudo service riak start </span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Times, Times New Roman, serif;">After these steps, riak will use the new EBS volume to store data instead the default storage space offered by the Ubuntu Server instance.</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<br /></div>
</div>
Adrian Copiehttp://www.blogger.com/profile/02438930394274756287noreply@blogger.com0tag:blogger.com,1999:blog-3917604409547043713.post-79266628985042009812013-01-02T18:47:00.003+02:002013-01-02T18:59:59.622+02:00<div dir="ltr" style="text-align: left;" trbidi="on">
<span style="font-size: large;">Delete all records from a riak database</span><br />
<span style="font-size: large;"><br /></span>
<b>Note</b>: Applies to riak 1.2.1 installed on Ubuntu Server 12.04 LT (hosted in Amazon EC2)<br />
<span style="font-size: large;"><br /></span>
The riak API allows to delete a bucket only if its content is empty. Sometimes (if you are performing various performance tests over the database, for example) the contained data is not important, so a script or program iterating through all the buckets and deleting all the content in a bucket is very time consuming.<br />
<br />
A most convenient solution is to delete the directories holding the data on the riak server. In order to do this, follow the steps below:<br />
<br />
1. Stop the riak server<br />
<br />
<span style="font-family: Courier New, Courier, monospace;">sudo service riak stop</span><br />
<br />
2. Change directory to the riak data<br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;">cd /var/lib/riak/bitcask</span><br />
<span style="font-family: Courier New, Courier, monospace;">sudo rm -r *</span><br />
<br />
<span style="font-family: Courier New, Courier, monospace;">cd /var/lib/riak/kv_nodes</span><br />
<span style="font-family: Courier New, Courier, monospace;">sudo rm -r *</span><br />
<br />
3. Restart riak service<br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;">sudo service riak start</span><br />
<br />
After these steps, the database is empty and riak is up and running.</div>
Adrian Copiehttp://www.blogger.com/profile/02438930394274756287noreply@blogger.com0tag:blogger.com,1999:blog-3917604409547043713.post-69015030067452333102013-01-02T11:01:00.002+02:002013-01-02T14:12:43.757+02:00<div dir="ltr" style="text-align: left;" trbidi="on">
<b><span style="font-size: large; vertical-align: baseline; white-space: pre-wrap;">Setting a riak cluster in Amazon EC2</span></b><br />
<b style="font-weight: normal;"><span style="font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><br /></span></b>
<b id="internal-source-marker_0.5443738694302738" style="font-weight: normal;"><span style="font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">Just some words related to a </span><span style="font-size: 15px; font-style: italic; vertical-align: baseline; white-space: pre-wrap;">riak 1.2.1</span><span style="font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> cluster installation inside the Amazon AWS network. The official information is very poor documented, some tips were discovered by trial and error, so maybe this post will help you if you are in the same position to build a riak cluster and fighting with poor documentation. </span><br /><span style="font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">I used Ubuntu Server 12.04 LT micro instances, 64 bits, from cost reasons.</span><br /><span style="font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">I use the following terminology in order to refer the nodes inside the cluster: Master node and a Slave nodes. The slave nodes will join the Master as they are needed. </span><br /><span style="font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">The first step is to install riak on every node as it is described </span><a href="http://developer.unitypark3d.com/ugamedb/manual/riak-on-amazon-ec2.html"><span style="color: #1155cc; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">here</span></a><span style="font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">.</span><br /><span style="font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">First of all, </span><span style="font-size: 15px; font-weight: bold; vertical-align: baseline; white-space: pre-wrap;">do not use</span><span style="font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> public IPs in configuring the individual nodes. Use the private IPs obtained inside the Amazon AWS network. </span><br /><span style="font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">Even I followed exactly the steps, when tried to start the riak service, I've fallen in the following error: </span><br /><span style="font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">Riak failed to start within 15 seconds,</span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">see the output of 'riak console' for more information.</span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">If you want to wait longer, set the environment variable</span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">WAIT_FOR_ERLANG to the number of seconds to wait.</span><br /><span style="font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">After examining the riak console, by issuing the </span><br /><span style="font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-family: Courier New, Courier, monospace; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">riak console </span><br /><span style="font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">command, the following information is useful :</span><br /><span style="font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">** Found 1 name clashes in code paths</span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">08:06:51.374 [info] Application lager started on node 'riak@XX.XXX.XX.XXX'</span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">08:06:51.418 [warning] No ring file available.</span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">08:06:51.457 [error] CRASH REPORT Process <b id="internal-source-marker_0.5443738694302738" style="font-family: 'Times New Roman'; font-size: medium; font-weight: normal; white-space: normal;"><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><</span></b>0 data-blogger-escaped-.165.0=".165.0"<b id="internal-source-marker_0.5443738694302738" style="font-family: 'Times New Roman'; font-size: medium; font-weight: normal; white-space: normal;"><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">></span></b> with 0 neighbours exited with reason: eaddrnotavail in gen_server:init_it/6 line 320<!--0--></span><br /><span style="font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">This means that another node is running in the memory and there is a clash in what concerns the port number.</span><br /><span style="font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">The solution is to issue a </span><br /><span style="font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">ps aux | grep riak</span><span style="font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> </span><br /><span style="font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">command and kill all the running riak processes, then restart the service. </span><br /><span style="font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">If the virtual machine hosting the riak service was stopped for various reasons, restarting it will change its IP so the configuration steps must be reiterated. However the ring information is no more valid anymore. In my situation, being a test installation, I didn't need the stored data so I have deleted the ring info by issuing the follow commands:</span><br /><span style="font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">cd /var/lib/riak/ring</span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">sudo rm -rf *.* </span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">In case of valuable data, the ring must be redone by performing:</span><br /><span style="font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">riak-admin reip <old_nodename> <new_nodename> <old_nodename> <new_nodename></new_nodename></old_nodename></span><br /><span style="font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">where <span style="font-family: Courier New, Courier, monospace;"><old_nodename></span> and <span style="font-family: Courier New, Courier, monospace;"><new_nodename></span></span><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><old_nodename> </old_nodename></span><span style="font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">are the names given in /etc/riak/vm.args</span><br /><span style="font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">Restart then the riak service. It should work now.</span><br /><span style="font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">If the settings were done on the Master slave, leave it as it is and go to the Slave node(s). For a Slave node, the next step is to join the current node to the Master node. This is done by issuing the command:</span><br /><span style="font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">riak-admin cluster join <master_nodename><master_nodename></master_nodename></span><br /><span style="font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">This is a staged command, it is only planned but not committed since a specific commit command isn’t issued:</span><br /><span style="font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">riak-admin cluster commit</span><span style="font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> </span><br /><span style="font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">Perform these steps for every Slave node in the cluster. </span><br /><span style="font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">Hope this post will help you to set a riak cluster in Amazon much faster and smoothly than me. </span></b></div>
Adrian Copiehttp://www.blogger.com/profile/02438930394274756287noreply@blogger.com1tag:blogger.com,1999:blog-3917604409547043713.post-17936175954699882032011-02-01T10:27:00.002+02:002011-02-01T10:29:36.040+02:00No comment<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgYRvvEX7Xl0IyJLXVRNz3RA-a6d4VsNXvQdup6Oc-xzYqgP5a-DoPGBKndEuB14fdSxtfIhTOlPDy7ozLle8y6fqgKCUNQw2r2wGE9-2ohzBdrXOhdymeRbqQYhU-6aAjU1Bl5lIHg5PO6/s1600/nosqlcv.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgYRvvEX7Xl0IyJLXVRNz3RA-a6d4VsNXvQdup6Oc-xzYqgP5a-DoPGBKndEuB14fdSxtfIhTOlPDy7ozLle8y6fqgKCUNQw2r2wGE9-2ohzBdrXOhdymeRbqQYhU-6aAjU1Bl5lIHg5PO6/s640/nosqlcv.jpg" width="449" /></a></div>Adrian Copiehttp://www.blogger.com/profile/02438930394274756287noreply@blogger.com0tag:blogger.com,1999:blog-3917604409547043713.post-75628850965467588842011-01-31T23:41:00.002+02:002011-01-31T23:46:21.578+02:00Storing Java objects with db4o<div style="background-color: transparent; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><div id="internal-source-marker_0.11792186717502773" style="margin-bottom: 0pt; margin-top: 0pt; text-align: justify;"><span class="Apple-style-span" style="color: #cccccc;"><span style="background-color: transparent; font-family: Georgia; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">These days I was looking for a database that should hold some native Java objects in a very easy to use way. Because when I documented myself about the NoSQL databases I have read a few words about Versant </span><a href="http://db4o.com/about/productinformation/db4o/"><span style="background-color: transparent; font-family: Georgia; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">db4o</span></a><span style="background-color: transparent; font-family: Georgia; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> database, I thought this is the time to know more about it. </span></span></div><span class="Apple-style-span" style="color: #cccccc;"><span style="background-color: transparent; font-family: Georgia; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></span><br />
<div style="margin-bottom: 0pt; margin-top: 0pt; text-align: justify;"><span style="background-color: transparent; font-family: Georgia; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><span class="Apple-style-span" style="color: #cccccc;">The idea behind this kind of database is to store the objects exactly as they exist in your application. No need for an additional relational layer which sometimes (often) maps the fields from a Java object to many fields in many table in a relational database. We have some immediately advantages from this approach:</span></span></div><span class="Apple-style-span" style="color: #cccccc;"><span style="background-color: transparent; font-family: Georgia; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></span><br />
<div style="margin-bottom: 0pt; margin-top: 0pt; text-align: justify;"><span style="background-color: transparent; font-family: Georgia; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><span class="Apple-style-span" style="color: #cccccc;"><span class="Apple-tab-span" style="white-space: pre;"> </span>- no need to install (or use) a RDBMS</span></span></div><div style="margin-bottom: 0pt; margin-top: 0pt; text-align: justify;"><span style="background-color: transparent; font-family: Georgia; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><span class="Apple-style-span" style="color: #cccccc;"><span class="Apple-tab-span" style="white-space: pre;"> </span>- the access speed to the database is increased due to the fact that we could bypass the relational layer</span></span><br />
<span style="background-color: transparent; font-family: Georgia; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><span class="Apple-style-span" style="color: #cccccc;"> </span></span></div><span class="Apple-style-span" style="color: #cccccc;"><span style="background-color: transparent; font-family: Georgia; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></span><br />
<div style="margin-bottom: 0pt; margin-top: 0pt; text-align: justify;"><span style="background-color: transparent; font-family: Georgia; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><span class="Apple-style-span" style="color: #cccccc;">The speed is also a strong point because under the hood the objects are stored in the form of graphs, which allow very efficient algoritms for reading, writing and searching.</span></span></div><span class="Apple-style-span" style="color: #cccccc;"><span style="background-color: transparent; font-family: Georgia; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></span><br />
<div style="margin-bottom: 0pt; margin-top: 0pt; text-align: justify;"><span style="background-color: transparent; font-family: Georgia; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><span class="Apple-style-span" style="color: #cccccc;">The database is written for two programming languages: Java and .NET. So, I have started with the download of the Java API and the binaries of the database, version 7.12. The archive is around 48MB and after extracting it, we have some directories containing the sources, the Object Manager Enterprise, which is a kind of administrative tool, and a directory containing some documentation and tutorials about how to install and how to use the database.</span></span></div><span class="Apple-style-span" style="color: #cccccc;"><span style="background-color: transparent; font-family: Georgia; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></span><br />
<div style="margin-bottom: 0pt; margin-top: 0pt; text-align: justify;"><span class="Apple-style-span" style="color: #cccccc;"><span style="background-color: transparent; font-family: Georgia; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The installation is very easy, and all that you have to do is to add in the classpath of your project (I used Eclipse to create a new project) of a library designed for your installed Java version. Personally I use Java 6, so I have chosen </span><span style="background-color: transparent; font-family: 'Courier New'; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">db4o-7.12.156.14667-all-java5.jar file.</span></span></div><span class="Apple-style-span" style="color: #cccccc;"><span style="background-color: transparent; font-family: 'Courier New'; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></span><br />
<div style="margin-bottom: 0pt; margin-top: 0pt; text-align: justify;"><span class="Apple-style-span" style="color: #cccccc;"><span style="background-color: transparent; font-family: Georgia; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Storing objects is very simple and intuitive. First we have to create an ObjectContainer, then use its methods which resembles with the one used in relational databases (I assume that many of us are familiarized with JDBC methods). So, for example we have an object Athlete and we want to store it in the database, we have to write (in an over-simplified way):</span><span style="background-color: transparent; font-family: Arial; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></span></div><span class="Apple-style-span" style="color: #cccccc;"><span style="background-color: transparent; font-family: Arial; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br />
<span style="background-color: transparent; font-family: Consolas; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">ObjectContainer db = Db4oEmbedded.openFile("db.dbo");</span><br />
<span style="background-color: transparent; font-family: Consolas; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">try{</span></span><br />
<div style="margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt;"><span style="background-color: transparent; font-family: Consolas; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><span class="Apple-style-span" style="color: #cccccc;">Athlete athlete = new Athlete("Haile Gebrselasie", "Berlin Marathon", "2:03:59");</span></span></div><div style="margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt;"><span style="background-color: transparent; font-family: Consolas; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><span class="Apple-style-span" style="color: #cccccc;">db.store(athlete1);</span></span></div><div style="margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt;"><span style="background-color: transparent; font-family: Consolas; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><span class="Apple-style-span" style="color: #cccccc;">db.commit();</span></span></div><span class="Apple-style-span" style="color: #cccccc;"><span style="background-color: transparent; font-family: Consolas; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">}catch(Exception e){</span><br />
<span style="background-color: transparent; font-family: Consolas; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><span class="Apple-tab-span" style="white-space: pre;"> </span>e.printStackTrace();</span><br />
<span style="background-color: transparent; font-family: Consolas; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><span class="Apple-tab-span" style="white-space: pre;"> </span>db.rollback();</span><br />
<span style="background-color: transparent; font-family: Consolas; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">}finally{</span><br />
<span style="background-color: transparent; font-family: Consolas; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><span class="Apple-tab-span" style="white-space: pre;"> </span>db.close();</span><br />
<span style="background-color: transparent; font-family: Consolas; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">}</span><span style="background-color: transparent; font-family: 'Courier New'; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br />
<span style="background-color: transparent; font-family: 'Courier New'; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></span><br />
<span class="Apple-style-span" style="color: #cccccc;"><span style="background-color: transparent; font-family: Consolas; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> </span></span><br />
<div style="margin-bottom: 0pt; margin-top: 0pt; text-align: justify;"><span style="background-color: transparent; font-family: Georgia; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><span class="Apple-style-span" style="color: #cccccc;">To search information in the database, we need queries. Here, the queries work over the instances of the same kind of objects. So, we could have Queries By Example which work based on an input template and returns all the objects that match all non-default fields of the template. A more general type of queries are the Native Queries, which in fact are the recommended way to search information. The lowest level type of queries are the so called SODA Queries (Simple Object Data Access) and they work directly with the nodes of the database graph. </span></span></div><span class="Apple-style-span" style="color: #cccccc;"><span style="background-color: transparent; font-family: Georgia; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br />
<span style="background-color: transparent; font-family: Georgia; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">For example retrieving all Athlet objects from the database using a Query By Example, first we have to define a prototype, then pass it to the query and wait for the result:</span><span style="background-color: transparent; font-family: Arial; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br />
<span style="background-color: transparent; font-family: Arial; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br />
<span style="background-color: transparent; font-family: Consolas; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Athlete proto = new Athlete(null, null, null);<span class="Apple-tab-span" style="white-space: pre;"> </span></span><br />
<span style="background-color: transparent; font-family: Consolas; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">List<athlete> res = db.queryByExample(proto);</athlete></span><br />
<span style="background-color: transparent; font-family: Consolas; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><span class="Apple-tab-span" style="white-space: pre;"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span></span><br />
<span style="background-color: transparent; font-family: Consolas; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">System.out.println("Name \t\t Race \t\t Time");</span><br />
<span style="background-color: transparent; font-family: Consolas; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">for(Athlete crtAthlete : res){</span><br />
<span style="background-color: transparent; font-family: Consolas; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><span class="Apple-tab-span" style="white-space: pre;"> </span>System.out.println(crtAthlete.getName() + "\t\t" + crtAthlete.getRace() + "\t\t" + crtAthlete.getTime());</span><br />
<span style="background-color: transparent; font-family: Consolas; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">}</span><span style="background-color: transparent; font-family: Arial; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br />
<span style="background-color: transparent; font-family: 'Courier New'; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></span><br />
<span class="Apple-style-span" style="color: #cccccc;"><span style="background-color: transparent; font-family: Consolas; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> </span></span><br />
<div style="margin-bottom: 0pt; margin-top: 0pt; text-align: justify;"><span class="Apple-style-span" style="color: #cccccc;"><span style="background-color: transparent; font-family: Georgia; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">I wanted to compare the speed of the db4o database vs a truly RDBMS (in fact I have MySQL 5 installed on my machine). For this, I have defined a table called ATHLETS in my RDBMS, following the schema:</span><span style="background-color: transparent; font-family: Arial; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></span></div><span class="Apple-style-span" style="color: #cccccc;"><span style="background-color: transparent; font-family: Arial; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br />
<span style="background-color: transparent; font-family: Consolas; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">create table athlets(</span><br />
<span style="background-color: transparent; font-family: Consolas; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><span class="Apple-tab-span" style="white-space: pre;"> </span>id int unsigned not null auto_increment primary key,</span><br />
<span style="background-color: transparent; font-family: Consolas; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><span class="Apple-tab-span" style="white-space: pre;"> </span>name varchar(50),</span><br />
<span style="background-color: transparent; font-family: Consolas; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><span class="Apple-tab-span" style="white-space: pre;"> </span>race varchar(50),</span><br />
<span style="background-color: transparent; font-family: Consolas; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><span class="Apple-tab-span" style="white-space: pre;"> </span>besttime varchar(50)</span><br />
<span style="background-color: transparent; font-family: Consolas; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">) engine=InnoDB;</span><br />
<span style="background-color: transparent; font-family: Consolas; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></span><br />
<div style="margin-bottom: 0pt; margin-top: 0pt; text-align: justify;"><span style="background-color: transparent; font-family: Georgia; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><span class="Apple-style-span" style="color: #cccccc;"> </span></span><br />
<span style="background-color: transparent; font-family: Georgia; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><span class="Apple-style-span" style="color: #cccccc;">Then, for 1000 records having the Name, Race and BestTime fields filled, the insertion time was 2109 miliseconds.</span></span></div><span class="Apple-style-span" style="color: #cccccc;"><span style="background-color: transparent; font-family: Georgia; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></span><br />
<div style="margin-bottom: 0pt; margin-top: 0pt; text-align: justify;"><span style="background-color: transparent; font-family: Georgia; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><span class="Apple-style-span" style="color: #cccccc;">For 1000 objects of type Athlet, with the same fields filled, the total insertion time was 106 miliseconds, so the db4o database is around 20 times faster than a relational database.</span></span></div><span class="Apple-style-span" style="color: #cccccc;"><span style="background-color: transparent; font-family: Georgia; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></span><br />
<div style="margin-bottom: 0pt; margin-top: 0pt; text-align: justify;"><span style="background-color: transparent; font-family: Georgia; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><span class="Apple-style-span" style="color: #cccccc;">To manage the records in the database, Versant provide an Eclipse plugin called Object Manager Enterprise (OME) which is very easy to install and then it gives you the opportunity to access and query the records in the database. </span></span><br />
<span style="background-color: transparent; font-family: Georgia; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><span class="Apple-style-span" style="color: #cccccc;"> </span></span></div><span class="Apple-style-span" style="color: #cccccc;"><span style="background-color: transparent; font-family: Georgia; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></span><br />
<div style="margin-bottom: 0pt; margin-top: 0pt; text-align: justify;"><div style="text-align: justify;"><span style="background-color: transparent; font-family: Georgia; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><span class="Apple-style-span" style="color: #cccccc;">I found this database very interesting and useful when need to store native Java objects. It has an amazing speed and also is very easy to use. The queries are checked directly in the compilation phase, so the parameters type too. The database is somehow schemaless. The object could change their structure then they are persisted as they are, no need for changes in the database layer (unlike RDMBS, where a change in the model implies the change in the database structure and more than that, a change in the SQL queries). It is embedded directly in the application, no need for an extra RDBMS to be installed somewhere locally or in the network. Being integrated in the application, it means that it is loaded in the same process. More than this, Versant db4o supports ACID transactions.</span></span></div><div style="text-align: justify;"><span style="background-color: transparent; font-family: Georgia; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><span class="Apple-style-span" style="color: #cccccc;"> </span></span></div></div><span class="Apple-style-span" style="color: #cccccc;"><span style="background-color: transparent; font-family: Georgia; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></span><br />
<div style="margin-bottom: 0pt; margin-top: 0pt; text-align: justify;"><span style="background-color: transparent; font-family: Georgia; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><span class="Apple-style-span" style="color: #cccccc;">I found also some cons, one of them from my point of view, is that for commercial applications or if you want support from the database provider, you have to pay for it. There is a scheme of licensing based on the number of processor cores and for a strong server, the price could escalate easily. </span></span></div></div>Adrian Copiehttp://www.blogger.com/profile/02438930394274756287noreply@blogger.com0tag:blogger.com,1999:blog-3917604409547043713.post-54065321965889623682011-01-29T19:46:00.007+02:002011-01-30T15:06:23.121+02:00Who's afraid of the NoSQL databases<div style="background-color: transparent; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><div id="internal-source-marker_0.3435000863391906" style="margin-bottom: 0pt; margin-top: 0pt; text-align: justify;"><span style="background-color: transparent; color: #cccccc; font-family: Georgia; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">No-SQL databases became a buzzword nowadays and myself being somehow involved in working with them, naturally I was asked by someone a couple of days ago what are they good for and to explain what’s the deal with them. Well, there is much to say about this subject, but below I will try to explain the essential things in few words. </span></div><span style="background-color: transparent; color: #cccccc; font-family: Georgia; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br />
<div style="margin-bottom: 0pt; margin-top: 0pt; text-align: justify;"><span style="background-color: transparent; color: #cccccc; font-family: Georgia; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The need for this kind of storage was initiated by the web applications that more and more dealt with bigger quantities of data which were sometimes distributed over more than one computer or were installed on servers which couldn't manage them at some point because of their physical limitations. In other words they couldn’t vertically scale, so they had to do it horizontally . For those who are not familiar with these terms, horizontally scaling it is the opposite for the vertical scaling in which we could improve the performance of a database by buying additional memory or increase the number of processors for the server which hosts the database. The horizontal scaling allows the increasing of the performance by adding supplementary nodes in a network, nodes that will host instances of the database. </span></div><span style="background-color: transparent; color: #cccccc; font-family: Georgia; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br />
<div style="margin-bottom: 0pt; margin-top: 0pt; text-align: justify;"><span style="background-color: transparent; color: #cccccc; font-family: Georgia; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Since 2009, a lot of different concepts of No-SQL databases appeared, all of them coming with advantages and disadvantages, but every one of them covering specific needs. Some of them are columnar databases (they store data in so called “columns”), other are key-value based and other store entire documents, they are called document store databases ,etc.</span></div><div style="margin-bottom: 0pt; margin-top: 0pt; text-align: justify;"><span style="background-color: transparent; color: #cccccc; font-family: Georgia; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> </span></div><span style="background-color: transparent; color: #cccccc; font-family: Georgia; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br />
<table style="border-bottom-style: none; border-collapse: collapse; border-color: initial; border-left-style: none; border-right-style: none; border-top-style: none; border-width: initial; text-align: justify;"><colgroup><col width="270"></col><col width="354"></col></colgroup><tbody>
<tr style="height: 0px;"><td style="border-bottom-color: rgb(170, 170, 170); border-bottom-style: dotted; border-bottom-width: 1px; border-left-color: rgb(170, 170, 170); border-left-style: dotted; border-left-width: 1px; border-right-color: rgb(170, 170, 170); border-right-style: dotted; border-right-width: 1px; border-top-color: rgb(170, 170, 170); border-top-style: dotted; border-top-width: 1px; padding-bottom: 7px; padding-left: 7px; padding-right: 7px; padding-top: 7px; vertical-align: top;"><div style="margin-bottom: 0pt; margin-top: 0pt; text-align: justify;"><span style="background-color: transparent; color: #cccccc; font-family: Georgia; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Key-Value Databases</span></div></td><td style="border-bottom-color: rgb(170, 170, 170); border-bottom-style: dotted; border-bottom-width: 1px; border-left-color: rgb(170, 170, 170); border-left-style: dotted; border-left-width: 1px; border-right-color: rgb(170, 170, 170); border-right-style: dotted; border-right-width: 1px; border-top-color: rgb(170, 170, 170); border-top-style: dotted; border-top-width: 1px; padding-bottom: 7px; padding-left: 7px; padding-right: 7px; padding-top: 7px; vertical-align: top;"><div style="margin-bottom: 0pt; margin-top: 0pt; text-align: justify;"><span class="Apple-style-span" style="color: #cccccc;"><a href="http://msdn.microsoft.com/en-us/library/dd179423.aspx"><span style="background-color: transparent; font-family: Georgia; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Azure Table Storage</span></a><span style="background-color: transparent; font-family: Georgia; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, </span><a href="http://scalien.com/"><span style="background-color: transparent; font-family: Georgia; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Keyspace</span></a><span style="background-color: transparent; font-family: Georgia; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, </span><a href="http://www.membase.org/"><span style="background-color: transparent; font-family: Georgia; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Membase</span></a><span style="background-color: transparent; font-family: Georgia; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></span></div></td></tr>
<tr style="height: 0px;"><td style="border-bottom-color: rgb(170, 170, 170); border-bottom-style: dotted; border-bottom-width: 1px; border-left-color: rgb(170, 170, 170); border-left-style: dotted; border-left-width: 1px; border-right-color: rgb(170, 170, 170); border-right-style: dotted; border-right-width: 1px; border-top-color: rgb(170, 170, 170); border-top-style: dotted; border-top-width: 1px; padding-bottom: 7px; padding-left: 7px; padding-right: 7px; padding-top: 7px; vertical-align: top;"><div style="margin-bottom: 0pt; margin-top: 0pt; text-align: justify;"><span style="background-color: transparent; color: #cccccc; font-family: Georgia; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Columnar Databases</span></div></td><td style="border-bottom-color: rgb(170, 170, 170); border-bottom-style: dotted; border-bottom-width: 1px; border-left-color: rgb(170, 170, 170); border-left-style: dotted; border-left-width: 1px; border-right-color: rgb(170, 170, 170); border-right-style: dotted; border-right-width: 1px; border-top-color: rgb(170, 170, 170); border-top-style: dotted; border-top-width: 1px; padding-bottom: 7px; padding-left: 7px; padding-right: 7px; padding-top: 7px; vertical-align: top;"><div style="margin-bottom: 0pt; margin-top: 0pt; text-align: justify;"><span class="Apple-style-span" style="color: #cccccc;"><a href="http://cassandra.apache.org/"><span style="background-color: transparent; font-family: Georgia; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Cassandra</span></a><span style="background-color: transparent; font-family: Georgia; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, </span><a href="http://hadoop.apache.org/"><span style="background-color: transparent; font-family: Georgia; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">HBase/Hadoop</span></a><span style="background-color: transparent; font-family: Georgia; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, </span><a href="http://aws.amazon.com/simpledb/"><span style="background-color: transparent; font-family: Georgia; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Amazon Simple DB</span></a><span style="background-color: transparent; font-family: Georgia; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></span></div></td></tr>
<tr style="height: 0px;"><td style="border-bottom-color: rgb(170, 170, 170); border-bottom-style: dotted; border-bottom-width: 1px; border-left-color: rgb(170, 170, 170); border-left-style: dotted; border-left-width: 1px; border-right-color: rgb(170, 170, 170); border-right-style: dotted; border-right-width: 1px; border-top-color: rgb(170, 170, 170); border-top-style: dotted; border-top-width: 1px; padding-bottom: 7px; padding-left: 7px; padding-right: 7px; padding-top: 7px; vertical-align: top;"><div style="margin-bottom: 0pt; margin-top: 0pt; text-align: justify;"><span style="background-color: transparent; color: #cccccc; font-family: Georgia; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Document Store Databases</span></div></td><td style="border-bottom-color: rgb(170, 170, 170); border-bottom-style: dotted; border-bottom-width: 1px; border-left-color: rgb(170, 170, 170); border-left-style: dotted; border-left-width: 1px; border-right-color: rgb(170, 170, 170); border-right-style: dotted; border-right-width: 1px; border-top-color: rgb(170, 170, 170); border-top-style: dotted; border-top-width: 1px; padding-bottom: 7px; padding-left: 7px; padding-right: 7px; padding-top: 7px; vertical-align: top;"><div style="margin-bottom: 0pt; margin-top: 0pt; text-align: justify;"><span class="Apple-style-span" style="color: #cccccc;"><a href="http://couchdb.apache.org/"><span style="background-color: transparent; font-family: Georgia; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">CouchDB</span></a><span style="background-color: transparent; font-family: Georgia; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, </span><a href="http://www.mongodb.org/"><span style="background-color: transparent; font-family: Georgia; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">MongoDB</span></a><span style="background-color: transparent; font-family: Georgia; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, </span><a href="http://code.google.com/p/terrastore/"><span style="background-color: transparent; font-family: Georgia; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Terrastore</span></a><span style="background-color: transparent; font-family: Georgia; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></span></div></td></tr>
<tr style="height: 0px;"><td style="border-bottom-color: rgb(170, 170, 170); border-bottom-style: dotted; border-bottom-width: 1px; border-left-color: rgb(170, 170, 170); border-left-style: dotted; border-left-width: 1px; border-right-color: rgb(170, 170, 170); border-right-style: dotted; border-right-width: 1px; border-top-color: rgb(170, 170, 170); border-top-style: dotted; border-top-width: 1px; padding-bottom: 7px; padding-left: 7px; padding-right: 7px; padding-top: 7px; vertical-align: top;"><div style="margin-bottom: 0pt; margin-top: 0pt; text-align: justify;"><span style="background-color: transparent; color: #cccccc; font-family: Georgia; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Object Databases</span></div></td><td style="border-bottom-color: rgb(170, 170, 170); border-bottom-style: dotted; border-bottom-width: 1px; border-left-color: rgb(170, 170, 170); border-left-style: dotted; border-left-width: 1px; border-right-color: rgb(170, 170, 170); border-right-style: dotted; border-right-width: 1px; border-top-color: rgb(170, 170, 170); border-top-style: dotted; border-top-width: 1px; padding-bottom: 7px; padding-left: 7px; padding-right: 7px; padding-top: 7px; vertical-align: top;"><div style="margin-bottom: 0pt; margin-top: 0pt; text-align: justify;"><span class="Apple-style-span" style="color: #cccccc;"><a href="http://www.objectivity.com/"><span style="background-color: transparent; font-family: Georgia; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Objectivity</span></a><span style="background-color: transparent; font-family: Georgia; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, </span><a href="http://db4o.com/about/productinformation/db4o/"><span style="background-color: transparent; font-family: Georgia; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">db4o</span></a><span style="background-color: transparent; font-family: Georgia; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, </span><a href="http://www.versant.com/"><span style="background-color: transparent; font-family: Georgia; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Versant</span></a><span style="background-color: transparent; font-family: Georgia; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></span></div></td></tr>
<tr style="height: 0px;"><td style="border-bottom-color: rgb(170, 170, 170); border-bottom-style: dotted; border-bottom-width: 1px; border-left-color: rgb(170, 170, 170); border-left-style: dotted; border-left-width: 1px; border-right-color: rgb(170, 170, 170); border-right-style: dotted; border-right-width: 1px; border-top-color: rgb(170, 170, 170); border-top-style: dotted; border-top-width: 1px; padding-bottom: 7px; padding-left: 7px; padding-right: 7px; padding-top: 7px; vertical-align: top;"><div style="margin-bottom: 0pt; margin-top: 0pt; text-align: justify;"><span style="background-color: transparent; color: #cccccc; font-family: Georgia; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Graph Databases</span></div></td><td style="border-bottom-color: rgb(170, 170, 170); border-bottom-style: dotted; border-bottom-width: 1px; border-left-color: rgb(170, 170, 170); border-left-style: dotted; border-left-width: 1px; border-right-color: rgb(170, 170, 170); border-right-style: dotted; border-right-width: 1px; border-top-color: rgb(170, 170, 170); border-top-style: dotted; border-top-width: 1px; padding-bottom: 7px; padding-left: 7px; padding-right: 7px; padding-top: 7px; vertical-align: top;"><div style="margin-bottom: 0pt; margin-top: 0pt; text-align: justify;"><span class="Apple-style-span" style="color: #cccccc;"><a href="http://neo4j.org/"><span style="background-color: transparent; font-family: Georgia; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Neo4J</span></a><span style="background-color: transparent; font-family: Georgia; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, </span><a href="http://infogrid.org/"><span style="background-color: transparent; font-family: Georgia; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">InfoGrid</span></a><span style="background-color: transparent; font-family: Georgia; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, </span><a href="http://www.systap.com/bigdata.htm"><span style="background-color: transparent; font-family: Georgia; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">BigData</span></a><span style="background-color: transparent; font-family: Georgia; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></span></div></td></tr>
</tbody></table><div style="text-align: justify;"><span class="Apple-style-span" style="color: #cccccc;"><span style="background-color: transparent; font-family: Georgia; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br />
</span></span><br />
<span class="Apple-style-span" style="color: #cccccc;"><span style="background-color: transparent; font-family: Georgia; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Other distinct characteristics of No-SQL databases are : they are </span><span style="background-color: transparent; font-family: Georgia; font-size: 11pt; font-style: italic; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">schemaless, </span><span style="background-color: transparent; font-family: Georgia; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">most of them are </span><span style="background-color: transparent; font-family: Georgia; font-size: 11pt; font-style: italic; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">open-source</span><span style="background-color: transparent; font-family: Georgia; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> with convenient licensing, are </span><span style="background-color: transparent; font-family: Georgia; font-size: 11pt; font-style: italic; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">distributed </span><span style="background-color: transparent; font-family: Georgia; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">and accessible through </span><span style="background-color: transparent; font-family: Georgia; font-size: 11pt; font-style: italic; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">simple APIs</span><span style="background-color: transparent; font-family: Georgia; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> which make them attractive for applications running in the Cloud, where they could spread over many network nodes. On the other hand, the No-SQL databases being usually distributed, they obey the Eric Brewer’s </span><a href="http://en.wikipedia.org/wiki/CAP_theorem"><span style="background-color: transparent; font-family: Georgia; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">CAP theorem </span></a><span style="background-color: transparent; font-family: Georgia; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">which says that for any distributed system, the following properties of the system cannot be provided in the same time:</span></span></div><span style="background-color: transparent; color: #cccccc; font-family: Georgia; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br />
<ul><li style="background-color: transparent; font-family: Georgia; font-size: 11pt; font-style: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div style="margin-bottom: 0pt; margin-top: 0pt; text-align: justify;"><span style="background-color: transparent; color: #cccccc; font-family: Georgia; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Consistency (all nodes see the same data at the same time)</span></div></li>
<li style="background-color: transparent; font-family: Georgia; font-size: 11pt; font-style: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div style="margin-bottom: 0pt; margin-top: 0pt; text-align: justify;"><span style="background-color: transparent; color: #cccccc; font-family: Georgia; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Availability (the system should have a response for every request in a specified amount of time )</span></div></li>
<li style="background-color: transparent; font-family: Georgia; font-size: 11pt; font-style: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div style="margin-bottom: 0pt; margin-top: 0pt; text-align: justify;"><span style="background-color: transparent; color: #cccccc; font-family: Georgia; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Partition Tolerance (the system should work despite the failures of some of its nodes) </span></div></li>
</ul><span style="background-color: transparent; color: #cccccc; font-family: Arial; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br />
<div style="margin-bottom: 0pt; margin-top: 0pt; text-align: justify;"><span class="Apple-style-span" style="color: #cccccc;"><span style="background-color: transparent; font-family: Georgia; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">So, being distributed automatically means that partitions are already created. From Brewer’s Theorem we know that we must trade between the other two remained properties. That’s the reason for we have databases that are more available than consistent or more consistent than available. For example </span><a href="http://cassandra.apache.org/"><span style="background-color: transparent; font-family: Georgia; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Cassandra </span></a><span style="background-color: transparent; font-family: Georgia; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">is highly available database, but eventually consistent, which means that in absence of any update operation during a limited period of time, all the nodes will hold eventually the same state. </span></span></div><span style="background-color: transparent; color: #cccccc; font-family: Georgia; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br />
<div style="margin-bottom: 0pt; margin-top: 0pt; text-align: justify;"><span style="background-color: transparent; color: #cccccc; font-family: Georgia; font-size: 11pt; font-style: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">In a next post I will talk a little about the data model in the NoSQL databases.</span></div></div>Adrian Copiehttp://www.blogger.com/profile/02438930394274756287noreply@blogger.com0tag:blogger.com,1999:blog-3917604409547043713.post-77960878281961780502009-10-29T11:05:00.000+02:002009-10-29T11:05:41.080+02:00So, who are the players?If you want to know who are the most important 150 Cloud Computing players, just click <a href="http://cloudcomputing.sys-con.com/node/770174">here</a>. It is an exhaustive list offered by <a href="http://cloudcomputing.sys-con.com/">Cloud Computing Journal</a>.Adrian Copiehttp://www.blogger.com/profile/02438930394274756287noreply@blogger.com0tag:blogger.com,1999:blog-3917604409547043713.post-18055268536965725842009-10-28T08:38:00.002+02:002009-10-28T16:37:06.150+02:00Google Über AllesLos Angeles becomes now the first city in the world that has decided to <a href="http://latimesblogs.latimes.com/technology/2009/10/city-council-votes-to-adopt-google-email-system-for-30000-city-employees.html">move the email system</a> for all its 30.000 employees to the Google Cloud by using Gmail, based on a contract of $7.25 million dollars. The fear for the data privacy in the Cloud was finally beaten, but some concerns still exists, regarding the fact that the security of the Cloud wasn't strongly proved yet .<br />
<br />
Google had fought against another very powerful competitor, <a href="http://www.microsoft.com/">Microsoft Corp.</a>, but the city concil unanimely approved their plan with a score of 12-0.Adrian Copiehttp://www.blogger.com/profile/02438930394274756287noreply@blogger.com0tag:blogger.com,1999:blog-3917604409547043713.post-9447612194842240232009-10-27T23:10:00.002+02:002009-10-28T12:52:13.797+02:00Image to GPSToday, surfing the net for some material related to Cloud Computing, I've landed on the one of the coolest algorithm of computer vision website I've heard. It is called <a href="http://graphics.cs.cmu.edu/projects/im2gps">IMG2GPS</a> and it was developed at the Carnegie Mellon University, by James Hays and Alexei Efros.<br />
<br />
What did those guys do?<br />
<br />
They have developed an algorithm that is able to extract some geographically information from pictures taken wherever the world, by leveraging the huge GPS tagged images database contained by <a href="http://www.flickr.com/">Flickr</a>. The location is determined taking into account some matches with color histogram, some textures, geometrical contexts and others, and not by considering people's skin color or clothes or other individualities related to humans. The algorithm is exemplified with three pictures, one representing the Notre Dame cathedral in Paris, one showing a street (probably a mediteranean landscape) and one representing an island, and after executing the algorithm, the proposed results are displayed, indicating the most probable, or even the exact location.<br />
<br />
Why is this algorithm interesting from the Cloud Computing point of view? Because it is a typically problem of iterating and processing huge quantities of data, a problem of parallelization of tasks and the current Cloud architectures offer the power and the tools for solving it.<br />
<br />
The tests were made using Flickr's images, but they are also a lot of other image providers, some of them hosting their databases in Cloud architectures and I am sure that soon we'll be able to determine ourselves the location of our interesting pictures. This is just the beginning in this branch of geographical computer vision, as the two authors named this research direction.<br />
<br />
The entire material could be found <a href="http://graphics.cs.cmu.edu/projects/im2gps/im2gps.pdf">here</a>.Adrian Copiehttp://www.blogger.com/profile/02438930394274756287noreply@blogger.com0tag:blogger.com,1999:blog-3917604409547043713.post-50065978925959612442009-10-23T17:38:00.002+03:002009-10-23T17:38:43.222+03:00The Fourth ParadigmVery interesting book: <a href="http://research.microsoft.com/en-us/collaboration/fourthparadigm/">The Fourth Paradigm: Data-Intensive Scientific Discovery</a>Adrian Copiehttp://www.blogger.com/profile/02438930394274756287noreply@blogger.com0tag:blogger.com,1999:blog-3917604409547043713.post-50973912996385991952009-10-22T20:25:00.000+03:002009-10-22T20:25:23.815+03:00This is the beginning...This is my very first blog and my very first post. Ever.<br />
<br />
Why was this blog born? <br />
<br />
Because I want to share my knowledge with people interested in <b><a href="http://en.wikipedia.org/wiki/Distributed_computing">Distributed Computing</a></b> and <b><a href="http://en.wikipedia.org/wiki/Cloud_computing">Cloud Computing</a></b>. I want to change ideas with all of you, guys, that have the same preocupation as I have, that want to make more to develop this idea, and maybe, one day, we'll have computing utility as easier as we have now electricity, gas, water and telephony.Adrian Copiehttp://www.blogger.com/profile/02438930394274756287noreply@blogger.com0