Cloudified

Now that we created a mobile app that can store data in the HTML5 local storage and a JSON REST backend, how do we make it scale? It’s time for true cloud storage.

Initially I thought of Amazon S3 as the storage backend. It’s highly reliable and scalable, bit only allows storing unstructured objects. We could store our data as blobs of JSON like we did in MySQL in the previous example. However, after reading Jurg van Vliet and Flavia Paganelli’s book on Amazon Web Services, it occurred to me that SimpleDB is a far better tool for the job. It has similar scalability and reliability characteristics as S3, but it stores simple structured data which can be queried. .. continue reading ..

One of the most useful principles in software development is the one that sounds the weirdest: idempotency. No, it doesn’t have anything to do with the e-mails you find in your spam folder. Idempotency is the characteristic of a piece of software to achieve the same result every time you run it.

It can be applied on various levels, the simplest of which is the level of the function. An idempotent function leads to the same result when you call it with the same parameters, no matter how many times you call it.

Being idempotent doesn’t mean that the function can’t have side effects: the result it’s supposed to achieve can very well be a file written to a disk or a record created in a database. The point is that when we call it multiple times, or from different start situations, the result will be once, and only once. There won’t be two similar files on the disk, nor two new records in the database.

.. continue reading ..

Developing software is hard. There’s lots of bits and pieces of information going around, and the head of a developer can only contain so much.

If you’ve ever developed a more than trivial piece of software, you’ve probably experienced what I like to call “The ‘huh??’ moment”: something impossible happens. You just added two columns to a table in your database, but when you run a query from code, they’re missing. You just fixed a particular bug, you deployed your code to production and it pops up again. You’re stepping through some code in the debugger, and your breakpoints aren’t being hit. The list goes on and on.

The perfectly normal reaction is to frown strongly, yell out “huh??” and doubt the technology you learned to trust. You’re seeing things that shouldn’t happen if everything worked okay, so something must be wrong, right? A bug in the database server? Did the deployment script fail? Is the debugger broken? You dive in and search for an explanation. And say “huh??” a couple of times more.

After years of saying “huh??” and hearing it a lot from other developers around me, my experience is that in all but a few cases you’re overlooking something when this happens. The tools you use have been tried and tested. It’s rare to stumble upon a bug in a database server.

It’s far more likely that one of the critical bits and pieces of information hasn’t correctly reached your head. Maybe you’re watching the wrong window or tailing a different logfile than you thought. Maybe you didn’t test your bugfix thoroughly enough, and in another situation it still occurs. Maybe your breakpoints aren’t being hit because that particular line of code is never executed.

Whenever you hear yourself utter “huh??” while developing, take a step back. Take some deep breaths, walk around, take some distance from the problem. That won’t hurt anyway, and I bet the solution will pop up as soon as you let go.

To make a mobile app really powerful, it has to interact with cloud services. We’ll set up a stateless service that speaks JSON, and show how our HTML5-based app can talk to it.

.. continue reading ..

Building a simple native mobile app for the major platforms wasn’t too hard. Neither was storing data on the mobile device. Will storing data in the cloud be just that simple?

Where to start?

Both a strength and a weakness of the cloud is that there are so many options to get stuff done. On the mobile device, your options as a developer are limited to what the device can do. In the cloud however, the options are almost limitless. That can be overwhelming.

We need some requirements to limit our options. We’ll look for a way to store data in the cloud that:

• is simple to set up and use.
• works well with JSON and JavaScript.
• is scalable in a fine-grained manner; not only upward, if demand for the application rises, but also downward: if our app has only three users, we don’t want it to use significantly more resources than those users require.
• is stable and robust, meaning it won’t fail if a single piece of (virtual) hardware of software fails.

.. continue reading ..

As software developers, we see an application as many different concepts.

Web app. Android app. iPhone app.

Database server. Web server. Application server.

User interface. Business layer. Data layer.

JSON. HTML5. PHP.

Your users experience your app as a single concept. Always available, personalized, on every device.

Facebook. GMail. Dropbox. Evernote. Twitter. Spotify. Delicious.

GMail on my laptop? It’s my GMail. GMail on my iPhone? Still my GMail. GMail on the public computer in the hotel lobby? Still, just the same, it’s my GMail. Many different technologies, pieces of hardware, networks. But all I experience is “my GMail”.

Everytime your app does what it promises, in a predictable, recognizable and consequent way, it adds a tiny bit to the trust your users have in it. When it does this over and over again, the concept of your app grows and grows more firmly in their heads.

Everytime when it doesn’t, it confuses them, alienates them, frustrates them. When it’s offline. When it does something unexpected and undesirable. When it looks and acts very different on a mobile device than on a web browser on the desktop. That puts a dent in their carefully grown trust.

Remember that, whatever it is you’re building. Your app lives in the heads of your users.

Now that we’ve established creating a simple example mobile app and running it on an Android phone, it’s time to let it do something more than generate a random number. The app has to connect: to the device it’s running on, and to one or more cloud-based services.

Since we’re building a reminder app, the first thing we’ll do is store and retrieve reminders.

Storage: on the device or in the cloud?

The storage capacity of different mobile devices varies wildly. So does the capacity and the stability of the network connection, a necessity to access cloud storage.

A good mobile cloud app makes this invisible to the user. It makes you forget that the capacity of your device is limited. It always works, it (almost) always has the latest data and the changes you make with it are available on other devices swiftly and transparently. To achieve this is one of the greatest challenges for mobile developers.

To make our app this smooth, we need mechanisms to:

• store and retrieve local data on the device;
• store and retrieve data in the cloud;
• synchronize both in a transparent, unobtrusive way.

Let’s start with storing local data.

.. continue reading ..

Looking around is a great way to learn, but doing is better. So let’s get with it: we’ll create a small mobile app in the upcoming series of posts.

The app will be:

• simple
• supporting the big mobile platforms (Android, iPhone, BlackBerry)
• making use of the unique features of today’s mobile devices (location, touch interface, accelerometer)

And although our primary goal here is to learn and to teach about the process of creating a mobile app, not the product it leads to, our app should have a purpose of some kind: We’ll create a location-based reminder app.

The app will allow the user to:

• define locations (work, home, shop, …)
• link reminders to these locations
• be reminded of these reminders when she gets in the proximity of the location, either on demand (“Is there something I should think about here?”) or automatically.

As we said before, development for mobile is hard. There’s a great variety of platforms and devices to take into account, and the capacity of developers is limited.

Fortunately, there are promising solutions to make mobile development more accessible: development frameworks. Initiatives in the area are popping up like mushrooms, and I must admit, those mushrooms look tasty.

We stumbled upon the following open source frameworks, all of them supporting all major platforms:

• Jo: a compact, well-documented UX library based on HTML 5, developed by a single person.
• Sencha Touch: a UX library also based on HTML 5, backed by a business. Seems to be more feature-rich and mature, but also heavier and more complex.
• PhoneGap: application development framework handling the specifics of different devices. For example, PhoneGap enables you to deploy your app to Android devices as a .apk, or create an iPhone app and submit it to the iPhone store, etc.
• Appcelerator Titanium: a generic framework supporting native development on multiple platforms. In contrast to HTML 5-based tech, Titanium allows using all native controls of the underlying platform for a very smooth user experience.

All of them have in common that the application logic is created in JavaScript. It’s nice to be able to use your existing JavaScript knowledge to develop a full-fledged mobile app.

Titanium look like it’s the odd one out, in a good way, because it’s the only framework supporting the native UX components of the different devices. Still, the HTML 5-based technologies feel like they’re better suited to what we’re doing here. We’ll start experimenting with PhoneGap, then add in Jo. Later on we’ll have a look at Sencha and Titanium.

Hello PhoneGap World

PhoneGap provides short and simple tutorials for creating and deploying your first app. The tutorials assume you start from a blank slate in terms of installed software on your development machine. Development with PhoneGap can be done on any machine running Windows, Linux or Mac OS X. We used the following components:

• Ubuntu Linux 10.10 (Maverick Meerkat) development PC
• HTC Hero phone running Android 2.2 (Froyo) as a testing platform.
• Eclipse IDE 3.6
• Android SDK

Walking through the tutorial for Android was a breeze. It’s a good idea to place a fresh copy of Eclipse in a folder of its own to prevent conflicts or version problems. Eclipse is very happy just living in a folder, without running an installer. The tutorial tells you the exact steps to configure Eclipse for Android development, so we won’t repeat those here. To know a bit more about the Android SDK you can have a look at the Android SDK documentation.

Testing your app is either done on an emulator (included with the SDK) or an actual mobile device. Here’s what our first “Hello World” looked like in the emulator, nothing more than a HTML page with a JavaScript alert:

You can interact with the emulator as if it were your phone. All the buttons are there, and using the mouse you can interect with the touch screen. Everything except true phone functions and multi-touch should be well testable.

After we “touch” our button, the alert shows:

What we see is nothing special; it’s a web page on a mobile device. What is special is that it’s deployed as a native mobile app, in a few easy steps. Based on the same source code we should be able to deploy it as an iPhone app.

Show us fancy controls, Jo!

Plain HTML with JavaScript wrapped in a mobile app is a nice start, but we want more. Jo offers a framework of mainly user interface components, and also some functionality to interact with data. Let’s explore.

To start with Jo, there’s an appealing video (12 minutes) and a set of samples included in the download. The samples worked well on our Android device. Forms, buttons, popups, animations, it’s all there. There was a major drawback however: slowness. Compared to native apps the controls were far less snappy.

Up to a certain level (or down, rather) that doesn’t have to be a problem. There’s always a performance tradeoff for generic, abstract components as opposed to native components. In some cases in the Jo samples, the performance on our first-generation Android device was too low for production usage, however.

This is likely the case with all HTML 5-based frameworks because the device has to work harder to parse the HTML and execute the Javascript. It’s important to test the performance of your HTML 5-based app on various devices, and choose wisely which options you’ll have to drop because the performance isn’t good enough.

A complete example: Randomness

To bring it all together we created an app using Jo and PhoneGap with a simple function: to generate a random number. The full source code is shown below, using some Jo controls for user interaction, and the joRecord component for data binding.

// Initialize jo.
jo.load();
 
// Setup a screen to show stuff.
var scn = new joScreen();
 
// Setup a joRecord for data binding.
var r = new joRecord({
    Number: null
});
 
// Create our view card.
var card = new joCard([
    new joTitle("Randomness"),
    new joCaption("Would jou like a new random number?"),
    new joDivider(),
    new joButton("Get me one!").selectEvent.subscribe(function() {
        r.setProperty("Number", Math.floor(Math.random() * 1000));
    }),
 
    // Bind the joRecord to some fields.
    new joGroup([
        new joLabel("Your number"),
        new joInput(r.link("Number")),
    ])
]).attach(document.body);

If you have a programming background, you should get the basic idea of how these lines work. For the full background on the Jo components and constructs, dive into the Jo Documentation.

This is what it looks like on an Android device:

A very practical aspect of developing in HTML 5 and JavaScript is that you can develop and debug your application in a capable web browser as well. Try opening the Randomness application with your browser. This way you can debug your app using the excellent FireBug or the Google Chrome debugger.

If you’re so inclined, you can download the Randomness APK to install the example on your Android device.

Final words (for now)

Deploying a simple mobile app using Jo, PhoneGap and some JavaScript was a breeze.

There’s a lot more to do to make the app actually do something useful, deploy it to other platforms and not unimportantly, get it in the application stores so users can download or buy it. And of course we haven’t even tried out Sencha Touch and Titanium yet. Stay tuned for more!

Cloud infrastructure makes it possible to dynamically scale your computing capacity with demand, both up and down. This allows for a zero-waste infrastructure. You never use more than you need, and you never pay for more than you need.

To make this work for your cloud-based application is easier said than done. It requires architectural choices and development actions that enable scalability. Each component needs to be loosely coupled to the rest of the infrastructure, and be stateless if possible. Developing for cloud scalability is a topic for a different article though. Probably another book or two. Today we’ll start from the comfortable (but hypothetical) idea that most services in the market are as elastically scalable as Werner Vogels ever imagined it.

What we will talk about in this post is the impact that the move toward scalable cloud infrastructure has on environmental resource usage. Power, water, clean air, those pretty scarce resources we all like to use so much. ICT is a huge factor in global energy usage and CO2 emissions. Data centers are resource-hungry beasts. To become a sustainable industry, ICT has to work towards using less and less resources. What I’m interested in is this: does a global move toward scalable cloud infrastructure have an impact on environmental resource usage? Does the promise of zero-waste help the ICT  industry decrease its negative impact on the world around us? Let’s explore.

Optimizing code means saving dough

A beautiful aspect of scalable cloud infrastructure is that it makes the relation between resource usage and cost very direct. Resource usage in a data center equals usage of the scarce resources of our planet. Establishing a 1:1 relationship between resource usage and cost means any save in resources is a direct cost saver. Saving resources through performance optimizations to any part of your application, your code, your infrastructure, leads to direct and immediate cost savings. Making an often-used piece of code more efficient not only makes your application faster, it means you use and pay for less computing capacity. This is a big step from the situation where a business would buy or lease a fixed amount of computing capacity for a long term: in that situation you would pay the same and use the same amount of energy no matter how well your application performs.

Investing in lowering your resource usage

Performance optimization also takes effort however. In many cases a lot of effort. Cost-wise there’s a break-even point for every optimization to a system. It’s like investing in a business opportunity. Investing means spending an amount of resources (usually money or labour) in order to have a chance to gain more resources in the long run. Depending on the success and the scale of the business, the investment will prove a success or a failure.

The fact that ICT resource usage has a direct relation to cost when using cloud infrastructure, means every performance optimization has the characteristics of a business investment. This goes even for very small performance optimizations. Managing your resource usage becomes managing money; resource usage becomes an economy. Politicians all over the world are trying very hard to turn environmental resource usage into an economy, using means like emission rights trading and whatnot, with only moderate success. In scalable cloud infrastructure this principle is implicit. Greater polluters pay more. There’s a financial incentive not to pollute and to waste for every user of cloud infrastructure.

Depending on the scale of the service, a possible optimization will either be a great opportunity to save costs in the long term, or a waste of resources in the short term. If your service only uses two virtual servers and you’re not expecting much growth, it’s hard to justify any optimization purely on the argument of cost. But if you’re using 100 computing instances at peak times which you can bring down to 80 by performance optimization, that offers a large margin to spend on making the service less resource-hungry.

The broader view

Besides the fact that we started from the non-existing, ideal situation that every cloud-based app is flexibly scalable in its resource usage, it’s not at all sure that this principle will make the ICT in this world use less resources overall. Cloud infrastructure may enable individual services to be as resource- and cost-efficient as possible, but the total amount of resource-hungry services will likely grow because computing capacity has become so ridiculously accessible to a large audience.

Furthermore, the principle works, but are the numbers right? The cost for computing capacity scales up and down with usage, but is it on par with the impact it has on the environment? Nobody knows. Data centers built today are much more energy-efficient than yesterday’s, rejoice! Too bad that the total amount of data centers built today surpasses that of former years, and the total amount of energy used by them is still rising.

So no, the advent of scalable cloud infrastructure will not save the environment by itself, and it’s likely to even increase the impact ICT has on the environment. It does however bring us closer to using less environmental resources, and it can contribute to decreasing overall energy usage indirectly by facilitating things like remote working. Scalable cloud infrastructure offers the potential to build every service in a resource-efficient way. Now all that’s left is making it so, and using the extraordinary possibility to easily access computing capacity in a responsible way.

Taxing the cloud-based polluter

A final word about the analogy with emissions trading. Emissions trading is a solution to the problem that certain business activities have hidden costs, which the business doesn’t pay for directly: an externality. Polluting is costly, because if possible it has to be cleaned up, and in many cases it causes irreversible damage to the environment. This cost should be paid for by the party causing the pollution. It can be realized in the form of a pollution tax. In a world where most computing capacity is delivered by a small amount of very large providers of cloud infrastucture, that’s far easier to realize than when every organization has a number of servers stuck in a cabinet.

Examples can serve as objects of inspiration and learning. To learn good development for the cloud, we can look at cloud services that work well. The first service we’ll explore is Spotify.

The good

Spotify has made me realize how I want “my” music collection to be. I want to have access to all kinds of music, when and where I feel like. But I don’t particularly want to hold a record in my hand. In fact, having to maintain a record collection is a burden. Records require space, insurance, cleaning, organizing, moving… you probably get the idea. I also want my music to be legal, and to support the artists that enrich my life with the product of their creativity and energy.

Spotify addresses these wishes in a very elegant way. Like most services, Spotify offers many things old and some things new. Old things include the technology to store music in a digital form, purchase it online, share it with friends and take it with you on a portable device.

The current ways to achieve these things were, in hindsight, less than optimal. To summarize: growing and maintaining an MP3 music collection takes a lot of effort over the years in finding, downloading and checking your files. Besides that, most sharing of MP3 files is illegal.

Any solution where the end user is responsible for storage of the files requires them to organize and backup the files. It takes effort to take part of your music collection with you. Whether it’s dragging files from one hard drive to another or configuring your latest portable music device to automatically synchronize with the latest version of your media player software: you need to take action every time you want to listen to new music on the road. If your brain is fallible like mine is, you’re going to forget this half of the time, and you’ll be stuck with the same tracks you’ve already heard a thousand times. Again. And moreover, you have better things to do than moving files around. Managing your music shouldn’t take a lot of your energy.

Enter Spotify. Spotify offers access to a very large library of music, for a fixed, low monthly fee, when and where you want. The service is responsible for maintaining the music, not the user. The only responsibility of the user is to select which music she wants to listen to. She can organize her music in playlists, star favourite tracks and share music with friends. In contrast to sharing music by exchanging files, the music lives on Spotify’s servers and is streamed to users.

We wrote that cloud apps need to go mobile. Spotify does this more than adequately. Mobile access is offered through apps for the various mobile platforms. Personally I have experience with the Android app, and almost purely positive. Spotify is synced seamlessly between all devices you use it on. Of course, from a software perspective, only syncing the users playlists and preferences, is a lot simpler than having to synchronize the music data.

For the user, Spotify feels like what is in my eyes a true cloud service. The cloud is always on, always available, and working uniformly on all ways you choose te access it. A cloud service is abundant on many levels. We’ve looked around for a term to define this, but haven’t found one yet. We’ll stick with “true cloud service” for now.

Developing for Spotify

Spotify is very interesting for users. But is it interesting for external developers? Two ways to interact with the service are offered on the Spotify Developers site: the Metadata API and libspotify.

The first is a RESTful API to be accessed over the web. It can be used to do most things a user can do with the Spotify apps except… play music. To do that, you need the second option, libspotify, which is a library for Windows, Linux an Mac you need to install locally. Hence with the current tools it’s not possible to develop a custom player on mobile platforms, only on PC’s. I haven’t experimented with the developer tools yet, but I’m eager to do so and will report my experiences.

The bad

A cloud service that offers access to a virtually limitless collection of music for a fixed, low fee is a Good Thing on many levels. There are downsides for the user though. The key term is “dependence”. I like what Spotify offers me as a user today. In fact I like it so much, that what is now my record collection in physical or digital form, I would love to turn into only a set of Spotify playlists and throw most of the originals in the bin. But that introduces a financial dependence and a risk.

To keep what I have, I’ll have to keep paying Spotify ad infinitum. And although Spotify is currently flourishing, it might not in two, five or twenty years (although I certainly wish them to). If I trust them to be responsible for my full music collection, that’s not a very safe idea. There are services which offer similar alternatives (supposedly Microsoft’s Zune offers a similar option), but how do I get my “record collection” from A to B? My conclusion is that Spotify needs an exit strategy for the user. The complete metadata of the collection should be exportable, and ideally importable to other online music services. The good news is that this is very well possible through the Metadata API. Any volunteers?

The ugly

The idea that Spotify sells is compelling, and they back it up with results. It works well for the user. However, reportedly it doesn’t work that well for the artists. Various artists have reported that they receive only meager compensation for being played on Spotify. This could partly be a matter of scale (as the paying user base grows, there’s a larger amount of money to spread), but there could be a structural flaw in the financial formula.

A key point is that we, the users, don’t know how much ends up with the artists. My advice to Spotify in this area is be as open as possible abount these numbers. How much money ends up with an artist for a track being played ten thousand times? By five thousand users? And if the compensation to artists is unreasonably low, consider offering more expensive subscriptions. Personally I would pay more if I know artists are compensated as well as, or better than when I buy their record in a store.

Wrapping up

As an example of a true cloud service, Spotify stands out from the crowd. It offers enormous value to its users for a very low fee. It’s accessible through many platforms in a simple, consistent way and its software works well. And although we don’t have hands-on experience, the developer tools look like they offer a powerful way to build your own integrations.

They definitely have some points to work on. A safe exit strategy for the users and a guarantee of good compensation to the artists should definitely be realized within a year or so. What I find interesting about those points is that they’re high-level problems which only rise up because most other things work so well. If the service had been unstable, or the music collection was too small, or the subscriptions had been too expensive, or the service was only accessible from computers with certain operating systems, probably very little people had known about Spotify. We’ll be following, and using, Spotify with great interest.