A spotlight on the gut at XSEDE'13

Possibly one of the most literally ‘in-depth’ talks I’ve attended at a computing conference came from Larry Smarr of the J. Craig Ventnor institute. He has got involved in biomedical research in the most personal way possible – by presenting his microbiome, or microbial profile, to be scrutinised in minute detail by researchers. The body has 10 times as many microbe cells as human cells. In fact, 99% of the DNA in your genome is in microbial cells not in human cells. “Topologically we’re a torus”, said Smarr. “Effectively, the bugs are all on the outside of your body, including your guts.”

Smarr’s interest in the microbiome started with increasingly severe but misdiagnosed stomach problems. Smarr was not impressed with the guesswork involved in treating his symptoms. He thought DNA sequencing should give a much more accurate picture of what was going on, eventually leading to a diagnosis of early Crohn's disease, an inflammatory bowel condition. With the cost of sequencing a genome fallen from 4 billion dollars per genome, to 4000 dollars, financing the research is not so much the issue – it’s picking out the microbial from the human, and then identifying what you have left. Research using the Gordon supercomputer at SDSC still found a significant proportion of DNA that was ‘unknown’. 

What is clear though is that Crohn’s disease has a devastating effect on the gut bacteria – the equivalent to a ‘mass extinction event’ in Smarr’s words. The traditional medical approach of waging war on the gut microbes using heavy duty drugs was not going to help – the better approach is to switch from full frontal attack to ‘gardening’. This means using new therapeutical tools to manage the microbiome and encourage ‘good’ bacteria. Diet is also an important factor. “Change the feedstock and you’ll change the shape of the garden,” advised Smarr. As he’s still donating all sorts of raw materials to the research programme on a regular basis, he should know.

(You can read more about microbial gardening on the CNN website – even the Economist has got in on the act with its cover page article ‘Microbes Maketh the Man’)

Biosciences Day at XSEDE’13 closed with a lively panel session featuring speakers from a range of high tech biomedical companies, including at least one lawyer. This is not as strange as it first sounds, because many of the issues affecting biomedical research come down to the ethics of sharing very personal data. “If you surprise a bioethicist with a question, the answer is always no” said Glen Orter of Dell. Alex Dickinson of Illumina Inc looked at the role of cloud computing in genomics, including two killer apps – whole genome assembly and variant calling, as well as single click sharing of very large data sets. His wish lists included cloud bioinformatics featuring openness and collaboration. “Interpretation is now the key, since analysis is no longer the bottle neck,” said Dickinson. This means thinking big and looking at phenotypes (how genes are expressed in the body) not just genotypes. “We want whole genomes and lots of them,” he announced.

Donald Jones of Qualcomm life talked about connected data, from wherever it orginates, inside the body, outside the body or from instruments. To bring in a Star Trek reference, this is the ‘tricorder challenge' to find simple to use multimeters for monitoring the body, like the blood glucose meters already used by diabetics. In the future, we’re likely to see increasing numbers of health related apps.
 

Darryl Leon from Life Technologies advised us to think holistically and also address the environmental costs of intensive computing – can we have low energy high energy computing? Nicholas Schork from the Scripps Institute said that sequencing genomes might be cheap at a few thousand dollars but interpretation can cost many times that amount. There are 4-6 million variants per person to explore using algorithms. “This is computationally intensive, and they may not do anything in the end or contribute to disease,” said Schork. There are a host of so-called idiopathic diseases where the cause is still unknown. Increasingly, the “exposome” will become important as well as the basic genome – the exposome is everything you’re exposed to in your life, including environmental factors. The more data we share on the exposome, theoretically the more information can be gleaned to lead to new treatments

The panel speculated that we might eventually see the equivalent of Facebook for sharing personal medical data – would you share your genome as blithely as as your holiday photos? But what about genetically identical twins where only one wants to publish their data, and the other prefers privacy?  Data might be compressed and encrypted – but this doesn't necessarily offer a full protection. Some people with Crohn's, like Larry Smarr, publish all their data in the hope of helping to find a cure. But others want total privacy – the challenge for the future will be to find a way to tread both these paths.
 

Biosciences Day at XSEDE'13

After the excitement of the XSEDE’13 kick-off next door to Comic Con, and the glamour of a 1940’s themed Chair’s dinner aboard the USS Midway, we moved into Biosciences Day. Touring the Midway, we squeaked across the blue and white checked lino of the mess deck (hearty hot meals were served 23 hours out of every 24) and ducked into the cramped confines of the sick bay with its three tier stacked bunk beds. The operating theatre and dental surgery were right next door to the machine shop. To the untrained eye, the equipment in all three looked broadly similar. (Given the gender bias of the crew, I’ll leave the identity of the two most requested elective operations to your imagination. They weren’t on the teeth). The basic nature of the medical treatment on offer to the crew however was a timely reminder of how far medicine has come in the last half century, particular now that high performance computing has joined the medic’s armoury.

David Toth, a Campus Champion at the University of Maryland Washington, talked us through the role XSEDE resources have played in finding potential drug treatments for histoplasmosis (a fungal infection), tuberculosis and HIV. In the days of the USS Midway, crew with a positive HIV test were immediately air lifted to shore, with a future behind a desk rather than at sea ahead of them. Toth’s group screened 4.1 million molecules using XSEDE resources, a task that would have taken 2 years on a single desktop PC. Some drug candidates are now being tested in the wet lab, with several promisingly showing the power to inhibit cell growth. “Supercomputers do a great job of outpacing the biologists,” said Toth. One enterprising feature of the work was that students each screened 10,000 molecules as part of their course, with one student striking lucky with a potential drug candidate.

Looking at biomedical challenges more broadly, Claudiu Farcas from the University of California in San Diego summarised some of the issues posed by data. For a start, there are many different kinds of data, from the genotype, RNA and proteome, on to biomarkers and phenotypes, right up to population level data, all with their own set of acronyms. Public repositories are often poorly annotated and are mostly unstructured as well as governed by limited and complicated data use agreements. “Researchers are encouraged to share but are not always enabled to do so,” warned Farcas.

A particularly thorny issue for biomedical data analysis is how to deal with sensitive personally identified information (PII). Researchers need to protect the privacy of patients by anonymising their data. It also needs to be cleaned up, compressed and aggregated so it can be analysed efficiently. But how best to do this? Bill Barnett, of Indiana University said earlier that biologists really don’t care what they use, as long as it works. Cloud computing can be tempting, but institutional clouds are often still at the early stages of being set up, and commercial sources might have “questionable” privacy protection.

The iDASH (Integrating Data for Analysis, Anonymization and SHaring) portal allows biologists to focus on the biology, not the data. It includes add-ons such as anonymisation of data, annotation, processing pipelines, natural language processing, and the possibility to preferentially select speedier GPGPU resource. According to Farcas, iDASH offers a secure infrastructure plus a platform for development, services and software, including privacy protection.

XSEDE’13 in San Diego – when super heros met supercomputers

There aren’t too many conferences where you meet Batman, Dr Who and the Wicked Witch of the West while still checking into the hotel. XSEDE’13 in San Diego this year overlapped with the famous Comic Con event right next door – so caped super heros marching past in the lobby was apparently part of the deal. Comic Con attracts over 120,000 participants every year, XSEDE slightly fewer at 700. But this is an ever rising number year on year, as project leader John Towns was pleased to point out. And I have a suspicion that the categories of ‘comic book nerds’ and ‘IT geeks’ are perhaps not entirely mutually exclusive sets…

XSEDE, the Extreme Science and Engineering Discovery Environment, is a National Science Foundation-supported project that brings together supercomputers, data collections, and computational tools and services to support science and engineering research and education. The annual XSEDE conference focuses on the science, education, outreach, software, and technology used by the XSEDE and other related communities worldwide.

The programme kicked off with a day of well attended tutorials – with a 200-strong host of students at the event, the training opportunities were well appreciated, as was the opportunity to showcase their work in poster sessions and competitions. Even younger participants were catered for by a morning robotics class, “Ready, Set, Robotical” which I was more than tempted to join.

Training is always a strong theme at XSEDE, and the challenges of providing online courses in parallel computing were discussed, as well as developing undergraduate and graduate programmes in computational science. Campus champions are the backbone of outreach on a regional basis, and XSEDE is now looking to expand the scheme into subject specific champions. This emerged as a theme for future collaboration between XSEDE and the European Grid Infrastructure in the Birds of a Feather session. EGI recently set up an EGI Champions scheme, including representatives from fields as disparate as life sciences, environmental sciences and maths. Other areas where EGI and XSEDE expect to work together include federated high throughput data analysis, federated clouds and user support. One use case already in progress covers HPC-HTC workflows for the computational chemistry community. This was one of the joint use cases that emerged in the call issued at the beginning of the year. So there's lots to watch out for, even now the caped crusaders have (mostly) left town.

Riding the data deluge on the shoulders of giants

The grid and the cloud are dead! Long live big open data! At least that seems to be the statement coming from a number of recent meetings.

Last week saw an exploratory meeting of the Research Data Alliance in Washington DC. Around 120 people from around the world had gathered to try and understand the blocking issues in allowing researchers to access and share data between disciplines, institutions and across countries. The issue here is not just around the pure issues of technical interoperability, but more broadly about building a community that can propose refine their technical work to reduce the barriers to a collaborative global data infrastructure.

The driving force behind this initiative (projects and funding agencies in USA and Europe) is not just about achieving technical operability, but about ensuring open access and exploitation of the big data sets being generated by researchers by others outside the generating community. An interesting example was presented by Chris Greer from NIST. He cited the how the release of NASA Landsat images in 2008 for unrestricted use had now created an estimated value of $935M/year to the environmental management industry. While it is not always necessary for the investments made by the public to yield economic returns – it cannot hurt!

However, the challenge in building any infrastructure is to balance the common needs against those that are specific to a particular science domain. Each research community will have developed its own vocabulary, its own metadata descriptions, its own data access services to expose the underlying data models. Where should the line be drawn between common and domain specific?

What are the common mechanisms that are needed to allow different research communities to collaborate and share data? While this is still work in progress, some consensus is emerging. For instance the need for persistent data identifiers that enable individual data sets and data objects to be described, discovered and located. Authentication and authorization is still needed when discussing open science data as funders like to know how the generated data is being used and it is possible that some data will be restricted to members of a collaboration for some of the time.

Leif Laaksonen from CSC described how within Europe the EUDAT (http://www.eudat.eu/) project is examining some of these technical issues with the recently started ICORDI (http://www.icordi.eu/) project now providing coordination and input into international activities such as the RDA. Andrew Treloar related how activities by the Australian National Data Services is working on helping scientists transforming data (generally unmanaged, disconnected, invisible and has a single user) to structured collections (that are managed, connected, findable and reusable) that can provide more value.

At this week’s Microsoft e-Science Workshop (that is co-located with IEEE e-Science and the Open Grid Forum 36 in Chicago this week) the focus on big data continued with sessions dedicated to Open Data for Open Science. Using environmental science as an example, with many examples drawn from NSF’s Earth Cube (http://www.nsf.gov/geo/earthcube/) initiative, the issues of data interoperability and reuse were again prominent.

The environment is an excellent example of how data reuse is needed across different domains in order to maximize knowledge discovery due to the inherent coupling between functions within the natural ecosystem and their impact within society. For instance, how do you ensure that satellite data can be coupled to land/sea/air observations collected independently over many years? How should the data coming out of the many instruments that make up an ocean observatory be integrated given the different manufacturers and their data formats?

The focus in this work is not so much on standard service interfaces but on standard data languages. Data markup languages encompassing the tracks of research vessels across the ocean, of the output from instruments, capturing the semantics of hydrologic observations are examples of many community driven initiatives. Organisations such as the Open Geospatial Consortium (OGC) – composed of representatives from academia and industry – play an important role for the environmental community due to the geospatial nature of many of their datasets and form a basis for much of the work that now takes place. The issue is about opening up your data for access not opening up your database!

Given the size and number of the environmental data sets generated from instruments and simulations, converting this data to information and knowledge provides many challenges. High Performance Computing can provide the raw simulation output and High Throughput Computing can help support ensemble studies to explore the sensitivity of the simulation. These local resources can be supplemented by capabilities accessed through grids (PRACE, EGI, OSG and XSEDE) and through commercial and publicly funded clouds.

While standards form one aspect of this discussion it is not the only , any standards need to encompass the variety of different use cases and users.

So has hype around big data grown to the point where it has now swallowed up the cloud which is still bloated on gobbling up the grid?

One of the challenges of big data is finding the infrastructure to analyse it! Here the cloud’s business model of the flexible and rapid provisioning of resources demonstrates its strengths. Creating the storage to handle the intermediate and final results generated from a HPC cluster provisioned on demand, demonstrates the need for a flexible model. As the data used in the analysis will need to be retrieved or placed in persistent data stores, issue such as authenticated and authorized access to these distributed resources becomes critical – a typical grid scenario.

In moving from one paradigm to another its important to not discard the experience and technology gained previously.

Getting ready for the Stampede at Texas Advanced Computing Center

It wasn’t exactly a stampede to get to the ballroom for yesterday’s post lunch talk (more of a gentle coffee-fuelled drift) but there was certainly a Stampede when we got there. James Gutowski of Dell was there to enthusiastically present the new system that the Texas Advanced Computing Center (TACC) will be deploying as part of the National Science Foundation's (NSF) "eXtreme Digital" (XD) program.

Named Stampede (you may have guessed), the new system will provide comprehensive computing and visualization capabilities for the open science community, helping  scientists to address challenging scientific and engineering problems. NSF is providing $27.5 million upfront and Stampede is expected to be up and running in January 2013. The system will replace Ranger (TACC) and Kraken (NICS).

Dell and TACC has gone for Linux/x86 and MIC processors (Many Integrated Core), having also assessed GPU, with MIC representing easier porting and more flexible programming options. Stampede is expected to have a peak performance of 10 PF with 14PB disk space, 250 TB RAM, 56 Gb FDR Infiniband interconnect and a substantial 200 racks of computer hardware. The base cluster is more than 6000 nodes and 100,000 cores but with a total concurrency of 500,000 cores.

January 2013 is racing around pretty fast, so the new data centre at TACC broke ground in February 2012 and the constructors are rolling in the racks as we speak. Find out more in HPCwire.

Field of Dreams: How are the XSEDE users using the infrastructure?

“If you build it, they will come.”  That’s a quote (or slight misquote actually) from the film Field of Dreams. The premise being that if you build something people really want (baseball field, website, supercomputer, cyberinfrastructure) they will find their way to it, and all will live happily ever after.

In the research and e-infrastructures field this is perhaps not the full story. It's true that potential research users are out there in large numbers, but matching them to the resources of their dreams often takes a lot of work, and not a little persuasion, outreach and training.

Yesterday’s talk by XSEDE’s David Hart of the National Center for Atmospheric Research on usage patterns for XSEDE resources was particularly interesting for those reasons – after a year’s operation and the transition from Teragrid, who are XSEDE’s users and how do they use the infrastructure?

From the official project rankings, it’s difficult to get a clear picture. Projects move up and down in the rankings over time, and the smeared out data doesn’t give much indication of usage. Hart decided to derive his own picture of the data, and looked at three factors: injection, continuation and completion ie how many projects were new in any one year and how long did they last?

In a given year, Hart discovered that around 40% were likely to be new, 40% continued beyond that year and 30% completed during the year. Larger projects were more likely to continue beyond the year, smaller projects often started and ended in the same year. The most likely duration is less than 6 months, corresponding to about 1000 projects. Around 63% lasted more than18 months.

When Hart turned his attention to the users of XSEDE, it turns out that projects are more stable than individual users – 48% of users in any given year were new, (meaning they had never submitted a job before) and about 30% carried on using the infrastructure into the next year. About 40% of users are graduate students however, so this perhaps correlates with the high turnover.

Out of the nearly 8400 users, 4000 users lasted less than 6 months. For about 2000 users, their first and last jobs were less than 15 days apart. This figure does not include the number of users who went to the trouble of logging in but never submitted a job at all. 

Hart was keen to point out that short lived usage of the infrastructure is not in itself something that you would want to avoid – this level of usage could still be exactly what the researchers and grad students need for their work, or to complete assignments and training. Comparing these patterns with the closest European equivalents to XSEDE would be interesting – PRACE, the supercomputing network has supported 103 projects, over 5 calls, representing more than 2.7 billion core hours. The European Grid Infrastructure, the pan-European grid computing infrastructure, has over 20,000 users across Europe and beyond and runs 1.2 million jobs a day. If you build it, and want 'them' to come, I guess you need to know your user.

Building bridges: XSEDE and PRACE announce joint call at XSEDE'12

As the temperature outside climbed to a baking 100 degrees, the atmosphere in the opening plenary for XSEDE’12 remained cool, calm and collected. John Towns, Principal Investigator for XSEDE talked us through the progress since the final Teragrid event last summer in Salt Lake City (also slightly on the warm side if I remember). Towns reminded us how XSEDE provides a variety of resources and services for multidisciplinary science, offering a comprehensive cyberinfrastructure through heterogeneous resources and services. XSEDE is based on the premise that the needs of the community are best served by a small number of centres – not a fully centralised model, but not fully decentralised either.

Since Teragrid, the focus has been on productivity and creating the environment which users need to be productive. The project has come round to the idea that it must define a solution which is designed to evolve, by identifying the greatest needs and starting there. As a 121 million dollar project, funding 120 full time equivalents over 17 partners, XSEDE also has to deal with the interesting problem that funding for the actual hardware is not included in this figure  - a not uncommon problem for Europe e-Infrastructures as well.

Nevertheless, the impacts of XSEDE's first year have been substantial, with the cyberinfrastructure supporting nearly 2000 projects for 8700 distinct users, funded across 32 National Science Foundation divisions and leading to 1800 publications. The transition from the Teragrid infrastructure to the XSEDE set up has proved seamless, with the user support service now supporting new architectures and new disciplines. This has also been supported by development of the human capital, a theme that is also very strong in Europe, through certificate and degree programmes. Stronger campus based communities have emerged, with successful users acting as bridgeheads into their communities. The user also now has a stronger voice within the project through the new User Advisory Committee “This is useful for those just trying to get their work done,” said Towns.

Campus bridging is gradually bringing ‘virtual proximity’ to the users, aiming to provide integration between campus workstations and the XSEDE cyberinfrastructure. These efforts are bringing their own challenges of course – bridging takes the technology in new innovative directions, but this can also lead to disruption when the main aim is to provide stable production services. “It’s an ongoing challenge to innovate in this environment,” says Towns. “We aim to support innovation both by partners within the project, but also within the community at large.”

XSEDE is building bridges not only between campuses and users, but also with other cyberinfrastructures. Open Science Grid is the US’s premier high throughput computing infrastructure and is now formally an XSEDE service provider, which should ultimately lead to better integration. XSEDE and its closest European equivalent, the supercomputing network PRACE, already run highly over-subscribed joint Summer Schools. Excitingly, they today released a joint call for Expressions of Interest from US and European researchers who would like to work together using both sets of resources. Dying to work with your colleagues across the pond (whichever side you happen to be on)? Find out more at https://www.xsede.org/xsede-and-prace-eoi and http://www.prace-ri.eu/Expressions-of-Interest.

Bridging from the eXtreme to the campus and beyond at XSEDE’12

After the excitement of the Higgs announcement at CERN recently, this week I am reporting for International Science Grid This Week from the XSEDE’12 event in the Windy City, aka Chicago. Apparently the term doesn’t just refer to the fresh breezes emanating from Lake Michigan, but also to the Chicagoans’ ability to debate up a storm. No doubt the organisers of XSEDE will be hoping for a similar quantity of discussion at this week’s event. XSEDE Principal Investigator John Towns said, "This conference series has evolved into an important meeting focused on the needs of the community supported by XSEDE, and it will cover a breadth of scientific, technical, and social aspects of cyberinfrastructure."

Today is tutorial day, and this morning I heard about selecting and using XSEDE resources for maximum productivity from Kimberly Dillman, Research Programmer and Campus Champion. The Campus Champions network was one of the strongest human capital assets to emerge from the Teragrid project which preceded XSEDE, and I am looking forward to hearing more about how this network has developed during the first year. Students are also a noticeably strong presence at this event – there are several tutorials aimed at students running today, and a designated Student Day on Wednesday, when students and educators are encouraged to come along and get a taster of the work that's going on. High school, undergraduate and graduate students are also invited to demonstrate the value that computational science adds to scientific discovery at the poster session. Last year the demos at this session were particularly engaging. 

 iSGTW is a media partner for XSEDE this year, along with HPCwire and I’m looking forward to catching up with some of the speakers: Richard Tapia of Rice University and recipient of the US National Medal of Science, John Towns of XSEDE, Thomas Eikermann of European supercomputing network PRACE, James Gutowski of Dell and Gayatri Buragohain, Feminist Approach to Technology, among others. It’s shaping up to be a varied week, concluding with Emmy Award-winning former producer of "60 minutes" Steven Reiner, who will discuss why it's important for scientists and researchers to make sure the public understands who they are and what they do.

As far as Chicago is concerned, it’s my second visit and I’m rediscovering some of the sights as well as following the conference. The locals seem to have a rather wry sense of humour – the spot where Mrs O’Leary’s cow supposedly started the great Chicago fire of 1871 by kicking over a lantern is now occupied by the fire training school. In the aftermath of the devastating blaze, Queen Victoria took the populace’s plight to heart and sent thousands of volumes of books to replace the public libraries lost to the flames. The city dwellers were duly grateful – but politely decided not to mention that before 1871, there were no public libraries in the City of Chicago.

So the stock of publicly available knowledge and data might have been given a timely boost as the city was rebuilt post-fire (not to mention a marked increase in the number of public libraries), but with tracks on science, technology, software and education, outreach and training in store, XSEDE'12 will bring together a further healthy body of knowledge, bridging out from Chicago, to the scientific community both here and abroad.