We managed to chase down workshop organiser David Colling for a quick chat. Watch below to see what David thought of the workshop so far.
Showing posts with label WLCG workshop. Show all posts
Showing posts with label WLCG workshop. Show all posts
Friday, July 9, 2010
Coming into the final straight of WLCG2010
First talk on the last day at a conference is always a daunting experience. People seem to agree with my assertions and conclusions that I gave in my summary of the experiment reports. ( I am really grateful to the VO contributors for their summaries as it made my job so much easier.) No one snored and no one complained I was talking rubbish which is always good. Looks like data management is the current bug issue and I am glad the VOs are aware of the issues for the T2 sites that thier need/want to use.
Thursday, July 8, 2010
Second day in London: experiment's day
After a quite intense first workshop day, ending up with a bunch of demonstrators (it looks as many people will be busy this summer) and a victory of Spain versus Germany on the world cup semifinals (Holland: be prepared), we got to the experiments jamborees today. I will try to highlight in this post some of the issues that were reported or discussed and draw my attention.
The first turn was for LHCb this time. We learned about how the current LHC running conditions are bad for this experiment. With that many protons per bunch and so few bunches, LHCb gets more pile-up than expected and less luminosity. They realised how bad this was when they tried to reprocess those busy events and found that the average reconstruction time was about twice the nominal and the stripping time... 80 times more!! They have adapted some parts of the software to throttle this and yesterday a new bunch of reconstruction jobs were sent. Fingers are crossed now to see if this time the events can be processed.
During the CMS slot, the guys of the red detector presented some quite impressive numbers of the rates at which their MonteCarlo re-digitisation and re-reconstruction workflows are capable of writing output data. In a big site like FNAL (6000 slots), these jobs can write up to 1PB of data every day! Not surprisingly, this has already generated some issues with the tape systems when trying to push this data stream into the archive.
In another presentation, an update of the model for planning the size of the different data flows at sites was presented. The numbers shown there for the bandwidth to the MSS surprised to some of the Tier1 sites in the audience since they looked pretty small (few tens of MB/s), specially if compared with the just shown avalanche of output data of the MC re-reconstruction. Modeling and planning is always tough in a fast evolving environment like the LHC. We will need to stay tuned to get the new calculations and plug them in the MSS purchase plans.
The ATLAS turn started with a really interesting presentation from Fernando. He showed the last "bells and whistles" of the ATLAS data management system. Plenty of monitoring stuff (yes! we love monitoring... you know) that shows which fraction of the used disk space are primary replicas, secondary replicas, etc. With all this information at hand they can do great things like a system that watches the disk occupancy and triggers an automatic deletion of "less important" data when 90% occupancy is reached.
A plot in this presentation showing the evolution of used disk at the different Tier1s versus time compared with the installed and pledged capacities generated some interesting remarks. Someone in the audience was worried because it was showing that ATLAS is only using about half of the pledged disk resources. Kors replied that actually the plot was showing that, when the LHC is taking data, ATLAS can fill up the disk very fast. This can be more worrying, indeed... if one thinks that we should survive with the current deployed resources until April next year. Luckily, Fernando's tools will make room for the new data to come... even if some secondary or tertiary replica needs to be trashed.
The first turn was for LHCb this time. We learned about how the current LHC running conditions are bad for this experiment. With that many protons per bunch and so few bunches, LHCb gets more pile-up than expected and less luminosity. They realised how bad this was when they tried to reprocess those busy events and found that the average reconstruction time was about twice the nominal and the stripping time... 80 times more!! They have adapted some parts of the software to throttle this and yesterday a new bunch of reconstruction jobs were sent. Fingers are crossed now to see if this time the events can be processed.
During the CMS slot, the guys of the red detector presented some quite impressive numbers of the rates at which their MonteCarlo re-digitisation and re-reconstruction workflows are capable of writing output data. In a big site like FNAL (6000 slots), these jobs can write up to 1PB of data every day! Not surprisingly, this has already generated some issues with the tape systems when trying to push this data stream into the archive.
In another presentation, an update of the model for planning the size of the different data flows at sites was presented. The numbers shown there for the bandwidth to the MSS surprised to some of the Tier1 sites in the audience since they looked pretty small (few tens of MB/s), specially if compared with the just shown avalanche of output data of the MC re-reconstruction. Modeling and planning is always tough in a fast evolving environment like the LHC. We will need to stay tuned to get the new calculations and plug them in the MSS purchase plans.
The ATLAS turn started with a really interesting presentation from Fernando. He showed the last "bells and whistles" of the ATLAS data management system. Plenty of monitoring stuff (yes! we love monitoring... you know) that shows which fraction of the used disk space are primary replicas, secondary replicas, etc. With all this information at hand they can do great things like a system that watches the disk occupancy and triggers an automatic deletion of "less important" data when 90% occupancy is reached.
A plot in this presentation showing the evolution of used disk at the different Tier1s versus time compared with the installed and pledged capacities generated some interesting remarks. Someone in the audience was worried because it was showing that ATLAS is only using about half of the pledged disk resources. Kors replied that actually the plot was showing that, when the LHC is taking data, ATLAS can fill up the disk very fast. This can be more worrying, indeed... if one thinks that we should survive with the current deployed resources until April next year. Luckily, Fernando's tools will make room for the new data to come... even if some secondary or tertiary replica needs to be trashed.
Jamie Shiers at the WLCG workshop
Video number two comes courtesy of Jamie Shiers who leads the grid support group in the IT department at CERN, and is responsible for the coordination of the worldwide LHC Computing Grid (wLCG) service.
Thanks for talking to us Jamie!
Thanks for talking to us Jamie!
Storage, storage, storage
The Grid does three things: Moves data, stores data and processes data. Arguably, the middle point, storage, is the most important*, as the other tasks would be pointless without it. Yesterday afternoon saw a session on storage issues, and as I had hoped for in my opening blog post, the session was indeed very forward facing.
A huge problem we have is data locality. Traditionally, we move data to sites, then run jobs at those sites. This can fail for a variety of reasons, such as
There was also talk of separating the tape system from the caching layer. I remain to be convinced how this can work in practice, but it is interesting to think that all tape operations may have to become operator-driven, rather than the system responding to files which need recalling automatically. That will require good tools to not make disk full and operators quivering wrecks.
A panel discussion on the future of Tier-1 storage followed. As far as I could see, everybody is basically happy with what they have now, and are experienced in operating their respective storage systems, but eyes must be kept open for the next technology to move to in time.
Finally, some presentations and demonstrations of prototypes to show new data management and access technologies were given. Two stood out to me. The first was given by Graeme Stewart of ATLAS fame, on their dynamic data placement work [1]. Essentially, their workflow management system can automatically notice which datasets are very popular, and replicate the data to Tier-2s which are not very busy. To be done next is to automatically re-schedule queued jobs, such that existing queued jobs can take advantage of the new data, rather than sit in a queue at a busy site.
Secondly, the ARC Caching model used by NorduGrid caught my eye [2]. This is a significantly different data management model to those used by gLite or OSG sites; the CEs pull in the data a job requires remotely, before running the job. What their caching systems does is cache this data locally on the front ends to save WAN transfers. Their systems has scaled to Petabyte-sized storage happily.
One can imagine a combination of these two strategies; downloading missing data at a site into a local cache, which is flushed with some Least Recently Used or Adaptive Replacement Caching strategy... Interesting stuff!
[1] http://indico.cern.ch/getFile.py/access?resId=0&materialId=slides&contribId=16&sessionId=2&subContId=2&confId=82919
[2] http://indico.cern.ch/getFile.py/access?resId=0&materialId=slides&contribId=16&sessionId=2&subContId=5&confId=82919
* Yes, I know this is a ridiculous statement, but I like making them.
A huge problem we have is data locality. Traditionally, we move data to sites, then run jobs at those sites. This can fail for a variety of reasons, such as
- Some files (out of many) are missing - jobs fail
- Site hosting data goes down - jobs disappear
- Some datasets are more popular than others - how to judge interest when subscribing data?
- Tape systems are slow - makes CPU efficiencies poor when waiting for recalls
There was also talk of separating the tape system from the caching layer. I remain to be convinced how this can work in practice, but it is interesting to think that all tape operations may have to become operator-driven, rather than the system responding to files which need recalling automatically. That will require good tools to not make disk full and operators quivering wrecks.
A panel discussion on the future of Tier-1 storage followed. As far as I could see, everybody is basically happy with what they have now, and are experienced in operating their respective storage systems, but eyes must be kept open for the next technology to move to in time.
Finally, some presentations and demonstrations of prototypes to show new data management and access technologies were given. Two stood out to me. The first was given by Graeme Stewart of ATLAS fame, on their dynamic data placement work [1]. Essentially, their workflow management system can automatically notice which datasets are very popular, and replicate the data to Tier-2s which are not very busy. To be done next is to automatically re-schedule queued jobs, such that existing queued jobs can take advantage of the new data, rather than sit in a queue at a busy site.
Secondly, the ARC Caching model used by NorduGrid caught my eye [2]. This is a significantly different data management model to those used by gLite or OSG sites; the CEs pull in the data a job requires remotely, before running the job. What their caching systems does is cache this data locally on the front ends to save WAN transfers. Their systems has scaled to Petabyte-sized storage happily.
One can imagine a combination of these two strategies; downloading missing data at a site into a local cache, which is flushed with some Least Recently Used or Adaptive Replacement Caching strategy... Interesting stuff!
[1] http://indico.cern.ch/getFile.py/access?resId=0&materialId=slides&contribId=16&sessionId=2&subContId=2&confId=82919
[2] http://indico.cern.ch/getFile.py/access?resId=0&materialId=slides&contribId=16&sessionId=2&subContId=5&confId=82919
* Yes, I know this is a ridiculous statement, but I like making them.
wLCG Photos
I've been sneaking around (pretty poorly) taking photos of the activities here at the wLCG workshop in Imperial College London.
Check 'em out:
They are all on one page here. If you are taking photos at the event and uploading to Flickr just tag them with wlcgworkshop and they'll appear in the slideshow in this post, in the sidebar and on the page linked just above this.
Later days,
Neasan
Check 'em out:
They are all on one page here. If you are taking photos at the event and uploading to Flickr just tag them with wlcgworkshop and they'll appear in the slideshow in this post, in the sidebar and on the page linked just above this.
Later days,
Neasan
Wednesday, July 7, 2010
First day - first try ...
OK, here we go. Blogging. First time for me as well. Anyway ...
I wanted to quickly list the most interesting things I found particular interesting today so that everyone can quickly look them up (on the indico page which can be found here: http://indico.cern.ch/conferenceOtherViews.py?view=standard&confId=82919):
- Tier-2 WAN networking was significantly improved by moving the Imperial College T2 outside the firewall. Who knew ... (http://indico.cern.ch/getFile.py/access?contribId=0&sessionId=0&resId=0&materialId=slides&confId=82919). But in general it is rather difficult to operate a university T2 because of the scattered infrastructure over the campus and of course the manpwer situation.
- A lot of discussion about the differences and different usage of GGUS and Savannah by the different communities. A good summary of the situation was that GGUS is a general ticketing system while Savannah is more used as a VO specific system
- Interesting fact from the Tier-1 presentation (http://indico.cern.ch/getFile.py/access?contribId=1&sessionId=0&resId=1&materialId=slides&confId=82919): Idle batch worker nodes use ~18tonnes CO2 /month (~£3K electricity). This is important as the usage of the Tier-1 sites by the LHC experiments is currently rather intermittend instead of flat. I think this has mostly to do with the amount of data we are currently procesing and the rush before the ICHEP conference.
- Ian gave a very good presentation about resource usage worth a read: http://indico.cern.ch/getFile.py/access?contribId=5&sessionId=0&resId=0&materialId=slides&confId=82919
The rest of the day was mostly spend with storage presentations and discussions and I leave that to James to comment :-)
Cheerio from London, Germany vs. Spain is starting in 1 hour, hopefully an interesting and exciting game.
OLI
I wanted to quickly list the most interesting things I found particular interesting today so that everyone can quickly look them up (on the indico page which can be found here: http://indico.cern.ch/conferenceOtherViews.py?view=standard&confId=82919):
- Tier-2 WAN networking was significantly improved by moving the Imperial College T2 outside the firewall. Who knew ... (http://indico.cern.ch/getFile.py/access?contribId=0&sessionId=0&resId=0&materialId=slides&confId=82919). But in general it is rather difficult to operate a university T2 because of the scattered infrastructure over the campus and of course the manpwer situation.
- A lot of discussion about the differences and different usage of GGUS and Savannah by the different communities. A good summary of the situation was that GGUS is a general ticketing system while Savannah is more used as a VO specific system
- Interesting fact from the Tier-1 presentation (http://indico.cern.ch/getFile.py/access?contribId=1&sessionId=0&resId=1&materialId=slides&confId=82919): Idle batch worker nodes use ~18tonnes CO2 /month (~£3K electricity). This is important as the usage of the Tier-1 sites by the LHC experiments is currently rather intermittend instead of flat. I think this has mostly to do with the amount of data we are currently procesing and the rush before the ICHEP conference.
- Ian gave a very good presentation about resource usage worth a read: http://indico.cern.ch/getFile.py/access?contribId=5&sessionId=0&resId=0&materialId=slides&confId=82919
The rest of the day was mostly spend with storage presentations and discussions and I leave that to James to comment :-)
Cheerio from London, Germany vs. Spain is starting in 1 hour, hopefully an interesting and exciting game.
OLI
Ian Fisk at the WLCG workshop
Here comes interview number 1 from the WLCG workshop here at Imperial College London.
Earlier today we chatted to Ian Fisk of CMS who told us about the issues his experiments faces, how he thinks the WLCG is going so far and his expectations for the future.
Earlier today we chatted to Ian Fisk of CMS who told us about the issues his experiments faces, how he thinks the WLCG is going so far and his expectations for the future.
wLCG So Far
Is it wLCG or WLCG? Do I care? Probably not.
The talks so far have been interesting, apparently ATLAS analyse a LOT of jobs on the grid. This was not really a surprise but the sheer number compared to the others was eye-opening. The case studies in the talks have been pretty interesting with so many just plain weird problems being at fault (who thought a network problem in Frankfurt would be so disastrous for data flow from Geneva to Taiwan?).
The general atmosphere in the room is quite jovial though with many people have been at different ends of all of the problems being discussed. James, Brian, Oliver and Gonzalo will give a better technical view from the day. However Manisha and I will be putting some of the big names in front of the camera later so if there is anyone in particular you would like to see there leave a comment on here and we will rope them in if we can. I think we have Ian Fisk up first as long as I can work the camera ...
Later days,
Neasan
The talks so far have been interesting, apparently ATLAS analyse a LOT of jobs on the grid. This was not really a surprise but the sheer number compared to the others was eye-opening. The case studies in the talks have been pretty interesting with so many just plain weird problems being at fault (who thought a network problem in Frankfurt would be so disastrous for data flow from Geneva to Taiwan?).
The general atmosphere in the room is quite jovial though with many people have been at different ends of all of the problems being discussed. James, Brian, Oliver and Gonzalo will give a better technical view from the day. However Manisha and I will be putting some of the big names in front of the camera later so if there is anyone in particular you would like to see there leave a comment on here and we will rope them in if we can. I think we have Ian Fisk up first as long as I can work the camera ...
Later days,
Neasan
Welcome to WLCG London
Having spent some amusing times being dragged kicking and screaming into pubs by Neasan wherever WLCG business takes us, I was clearly deemed fit to tell the world about the goings on at the WLCG workshop in London.
My interest is in storage management. Ideally, it should just work; I am a physicist and the computing is a tool to allow us to "do physics". However, with a global computing system of this scale and complexity, there will always be issues. The situation now is much better than if I were writing this post a couple of years ago - the software is maturing, the community expertise is greater than ever, and we have the sharp focus of real LHC data to keep us on our toes.
However, there is still much room for improvement. The experiments (well, the one I work on, CMS, at least) continue to find new and novel ways to abuse the computing systems, and we have to respond to those. New scales of workloads show up new problems. Systems that weren't deemed critical suddenly become the weakest link. Users complain when their data isn't on tape in 5 seconds flat. This list goes on, and the tools need to be responsive.
I'm hoping this workshop will be a look back and a look forward. We've come on a long way and everyone here should be proud of that. However, we have a long way to go - the scary volumes of LHC data are yet to come.
My interest is in storage management. Ideally, it should just work; I am a physicist and the computing is a tool to allow us to "do physics". However, with a global computing system of this scale and complexity, there will always be issues. The situation now is much better than if I were writing this post a couple of years ago - the software is maturing, the community expertise is greater than ever, and we have the sharp focus of real LHC data to keep us on our toes.
However, there is still much room for improvement. The experiments (well, the one I work on, CMS, at least) continue to find new and novel ways to abuse the computing systems, and we have to respond to those. New scales of workloads show up new problems. Systems that weren't deemed critical suddenly become the weakest link. Users complain when their data isn't on tape in 5 seconds flat. This list goes on, and the tools need to be responsive.
I'm hoping this workshop will be a look back and a look forward. We've come on a long way and everyone here should be proud of that. However, we have a long way to go - the scary volumes of LHC data are yet to come.
My first blog for gridcast
Well I am ready for the off. The WLCG conference is going to be the first conference I will be blogging for GridCast at. With my work with the UK Tier2s, I am hoping to get alot from the results of last's week's WLCG Data Management meeting in Amsterdam. Plus I am also demoing in the session later today; so hopefully that will go well.
Tuesday, July 6, 2010
WLCG workshop on GridCast from tomorrow!
Tomorrow we're heading over to the WLCG workshop at Imperial College London, to cover the event live on GridCast!
We've got a great team of bloggers to keep you updated with all the latest experiment and grid related news. So from 7 - 9 July stay tuned to the blog to stay in the (LHC) loop!
In the meantime you can have a look at the workshop programme here.
Subscribe to:
Posts (Atom)
Posts
