Opening the Cloud in Paris

Today I am in a very chilly Paris for a meeting with the complex systems community, to talk about clouds. The event is organised by the ASSYST project, coordinated by Jeff Johnson of the Open University and is funded from the FET Proactive initiative Science of Complex Systems for Socially Intelligent ICT (COSI-ICT). The COSI-ICT programme has four integrated projects, supported by ASSYST. The work packages of ASSYST are closely aligned with the activities of the Complex Systems Society.

The community is dealing with the challenges of cloud computing that many other scientific disciplines are also getting to grips with – legal issues about who owns the data, the currently limited interoperability between clouds, concerns about security of the data and scalability, needing resources that react in real time to demand. This community is also particularly interested in the new social auto organisation phenomena that might emerge from using clouds.

A new service that is emerging is brokering of spare cloud computing capacity, through sites such as SpotCloud. Sellers can offer their spare capacity to buyers when they know it will be available, for example overnight or on a quiet Sunday afternoon. Buyers can benefit from lower prices, and a single point of sale.

Another aspect that emerges is the need for a new generation of engineers. These paragons of complexity will need to be able to cope with warehouse scale computing, dealing with 10 million servers at a time, programming for processors with hundreds of cores and a flow of data at the scale of petabytes or exabytes.

StratusLab aims to create an open source and comprehensive “private cloud” distribution, something which doesn’t exist at the moment. Cal Loomis, the project coordinator, outlined some of the advantages of clouds, such the customised environment, rapid access via a simple API plus complete control to install what you want on a ‘pay as you go’ model. The disadvantages include non-standard interfaces and effective cloud lock-in, plus creating a new virtual machine is difficult, there are lots of things that can go wrong along the way.

In StratusLab’s view, grids and clouds are complementary to each other and they are using the best elements of both. The grid’s uniform security model based on PKI certificates means that users don’t need local accounts to access resources. Grids also share resources, algorithms and expertise in a way that brings distributed resources together to make them look like one unified resource. From clouds, StratusLab brings in the ability to allocate resources dynamically, which grids are weak on, plus the capacity to customise environments. StratusLab has set up a grid resource centre and runs the grid services on top of a cloud API – users can access the resources at either level. From there, they can also bring in public clouds as well, and the users see no difference from using a normal grid resource centre. There are still challenges to solve however – data management, maintaining the illusion of “infinite” resources, building trust between systems administrators and users and distributing resources fairly. 'Pay as you go' for example is more complex in an academic environment, when the funders are usually remote from the researchers doing the work.