"Poor man's" application virtualization on Linux.

an update.

Francesco Prelz

INFN, sezione di Milano

INFN-Condor support group

Summary

• Perspective: be able to offer the good services of the INFN Condor pool (or other Condor pools) to INFN's institutional users: mostly from LHC experiments.
• Need to easily migrate their applications: find a workable replacement to the static linking currently used for the Condor standard universe...
• ...but short of packaging and supporting virtual machines of various flavours.
• ...and of any superuser-privileged operation.
• The reality check comes from High Energy Physics (CERN LHC Atlas Experiment event reconstruction) code.
• Several ways were explored. I focus on an approach based on Parrot. There may still be better ways to do it!

If I were a rich man...

• If I were a rich man I could learn and use some hypeful technology.
• VM images (or compressed 10 GB disk images) pre-packaged with everything an LHC applications needs can be downloaded from Predrag Buncic's CernVM project.
• Anything that's missing is fetched transparently from the network using a FUSEd filesystem. Currently (and interestingly enough) Parrot and its cache sit behind FUSE.
• The problem is as usual where data to handle these cache-misses can be efficiently grabbed. What is proposed is to set up some worldwide hierarchy of HTTP caching proxies.

But I'm a poor man. Why ? (1)

• Poor people own (at most) their PC.
• Poor people can't run as root anywhere else.
• Poor people don't get heard by system administrators.
• Poor people can't pay for present or future VM technology licenses.
• Poor people have to feel at home everywhere and anywhere.

Why poor man's ? (2)

• Poor people hate wasting resources:
  

Why poor man's ? (3)

• Poor people are usually left alone.
• Poor people must also be careful, as they run a high risk of being occasionally steamrolled:
  

ID card of our application.

• One release of Atlas software, as installed, includes about 9.5 GBytes of material: not light-as-a-feather.
• Outside the Atlas software install area, 680 MBytes of user code and libraries (250 MBytes gzipped) are accessed.
• Scanning the process trace of a code run (reconstruct one top quark event) shows that:
  • 3998 files are accessed
  • 6452 successful open calls are made
  • 44604 open calls failed (99.6 % ENOENT)
  • 43890 successful stat calls are made
  • 45598 stat calls failed (99.8 % ENOENT)
  • Grand total of 136 MB I/O (I: 125 O: 11)
  • 49% of input comes from three files (whose size is 180 MB total)
• A selection of the files that were actually accessed, plus enough files to make the code manager happy adds up to a whopping 3146 MBytes total (961 MB gzipped).
  Moving them does not appear to be the correct approach.

First try

• First try: combine chroot and NFS:
  
• But chroot(2) and mount(2) are privileged. And mounting NFS on Linux is via a helper program :-(

The Parrot approach

• Papa Todd says: use Parrot and Chirp!
  
• With a little bugfixing (see CCtools ML for details), it works!

Figures for Atlas reco - NFS

• i686 native code running on x86_64 machine.
• Data access pattern:

  N. evts	Wall time	CPU time	Transferred packets	Transferred bytes
  1	6m 25.518s	4m 43.146s (73.4 %)	497,029	486.6 MB	Full NTOP report
  10	11m 52.928s	10m 6.917s (93.5 %)	524,680	508.8 MB	Full NTOP report

• Mostly at full MTU:

Figures for Atlas reco - Parrot

• Data access pattern:

  N. evts	Wall time	CPU time	Transferred packets	Transferred bytes
  1	8m 1.372s (20 % more than NFS)	5m 23.484s (67.2 %) (14 % more than NFS)	1,261,978	750.0 MB	Full NTOP report
  10	16m 48.385s (29 % more than NFS)	12m 27.595s (74.1 %) (19 % more than NFS)	1,314,194	770.5 MB	Full NTOP report

• Only 36% at full MTU:

Conclusions

• Atlas reconstruction can be parachuted and happily run onto a remote node with no previous configuration.
• Not getting help from the administrator by running parrot out of the box costs us an O(30%) time penalty. Need to work on optimising this.
• Real life High Energy Physics applications seem to require either multi-gigabyte software distributions to be locally available, or network bandwith on the order of 6 Mbit/s per active job, and low latency. Yes, the data problem is still with us.
• Could we think of moving this application to the new, improved standard universe and replace chirp_server with condor_shadow and friends ?
• Can a self-extractable Parrot cocoon be outcome of some future instance of condor_compile?

Thank you

• Thank you for your patience.
• Questions or comments?
• and remember: always watch for the steamroller close to you.