Mine the Harvest

I have officially started my new position at Oak Ridge National Laboratory outside of Oak Ridge, Tennessee. I am working in the Information Technology Services Division, which is part of the Computing and Computational Sciences Directorate. My position is that of Unix / Linux system administrator, working primarily in maintaining systems and infrastructure supporting the Climate Change Science Institute.

The lab has a number of core focus areas including environmental and climate science, materials science  (including new facilities for nanophase research), and is a world leader in neutron science at the Spellation Neutron Source. ORNL is also the home of the National Center for Computational Sciences, and the home of Jaguar and Kraken, currently the first and third most powerful supercomputers in the world. ORNL also supports national security projects, nuclear non-proliferation work,  nuclear science and engineering, fusion power and alternative energy sources as well as being a leader in computational biology.

These diverse research areas all typically heavily utilize Linux and Unix for a number of scientific applications and research tools. Of course, the supercomputing center relies on Linux almost exclusively as the Cray systems, and various support clusters, run various distributions of Linux.

Spallation Neutron Source (SNS) at ORNL

The multi-disciplinary environment of the lab makes for a very interesting work environment. I am currently supporting researchers in their work in climate sciences, for example by maintaining the systems used for the ORNL Distributed Active Archive Center (DAAC), and the Atmospheric Radiation Measurement Program (ARM) systems. Additionally, I will be working more with biological science systems as time goes on.

Jaguar Cray XT5

Eventually, I would like to explore my interests in high performance computing, both in the computational needs for Environmental Sciences, and otherwise. I’m increasingly interested in GPU computing, and would like to work with developing and supporting compute environments which use NVIDIA CUDA and OpenCL which can help realize dramatic performance increases in massively parallel computing.

Fall Colors at ORNL

The lab is a world-class facility and is the showcase of our national laboratory system. The campus grounds are quite nice. The lab is surrounded by trees, and is on a huge reservation. The drive to work is beautiful. Here is a picture of the fall colors behind the Joint Institute for Biological Sciences building.

Everyone  at the lab has been very easy to work with; in fact, I would have to say the people make up one of the best aspects of working at the lab. It has been a pleasure getting to know my new colleagues and customers.

Our move to Tennessee went very well, and Sydney and I are planning to close on a home up here this month. It’s been a big change for us, but we were very much ready to leave Florida. The changing seasons, smaller urban area, and ready access to the Great Smoky Mountains National Park are much more to our liking than the urban sprawl of Tampa bay. So far we have been on three hikes to the Smoky Mountains and each of them was breathtaking. We once again live someplace where we can truly enjoy the outdoors. (We got a bit spoiled living in Colorado and Alaska!)

Moving up here was a big adventure, but I am happy to say that everything is going very well. The relocation assistance provided by the lab has been tremendously helpful and we are starting to feel very much at home here. I’ll post some pictures of our hikes soon, and do some updates from time to time on what projects I am working with at the Lab and other interesting news.

(Update: I’ve written the Wiki now and you can find it here.)

So I’ve been playing around with NVIDIA’s CUDA Toolkit and SDK and reading this book that just came out “CUDA by Example”. (See this Linux Journal article for some general back ground on CUDA.)

This is pretty interesting (and fun) stuff. So far I’ve gotten the toolkit installed and compiled the sample code projects in the SDK. I have to use my workstation at work as (sadly) my ancient ass NVIDIA card at home is not CUDA capable and my system is so old it is really just a type writer at this point.

Running just the sample CUDA projects is pretty impressive though. My old school GeForce 8800 at work has a meager 32 cores. (The card I just ordered has 96, and the most recent Fermi architecture ones have 448 and 6GB – yea, holy crap.) However, with only 32 cores it renders the particle, smoke and other samples quite nicely. These are rendered in 3D, allowing you to change perspective, and sometimes alter the particle density, light source, etc.  They are just examples showing how to run general purpose code (and render results in OpenGL) on the GPU.

So then I thought – I would like to forward the rendered OpenGL images to my workstation at home. Alas, SSH, VNC, et. al all actually send the OpenGL to the client to be rendered. Enter VirtualGL, which does just what you would think – it attaches a loadable module to an executable which intercepts OpenGL requests and send them to the local GPU for rendering, putting the results into a pixel buffer which then gets sent to the client. Wow. You can run your rendering in a data center and forward the results anywhere. That’s nifty!

Here is a picture of the CUDA smoke particles sample, being rendered at work, and forwarded via SSH X Forwarding with rendering handled by VirtualGL. (I think Geek Level just dinged.)

CUDA with X Forwarded OpenGL

I’m going to write up a wiki on this stuff, but if you have any recent NVIDIA card (8800 or newer) , even a mobile one, you likely have the capability to run CUDA. If you are interested you can download the Toolkit and SDK and have at it. (Update: I’ve actually written the Wiki now and you can find it here.

After testing the CUDA waters I have to say I am very impressed. I can easily see CUDA being integrated into many applications in the near future. There are already plugins to offload MS Excel formulas onto the GPU, run Mathmatica on it, etc. The integration of multi core CPUs, in even consumer class hardware, combined with the capabilities of GPUs very well might make things go from a “multi core revolution” in consumer computers and into a “massively parallel revolution”. In fact, it’s already happened. Massively parallel computing is now capable on your notebook, it just has not been fully exploited yet. (Think of the cryptographic fun you could have.) Of course once Apple catches on, it will be called iMassive.

This is just all pretty fascinating to me. My first computers had 4Mhz Motorola CPUs, 32K of memory, etc. and now I just bought a 98 core GPU which I can write code in CUDA C and run general purpose computations on. Wow. And I’m getting ready to get an i7 too – it’s all just a bit dizzying. But very fun.

If you have not checked out CUDA, I highly recommend it. It’s a good opportunity to learn more about massively parallel computing, which I think we are going to be seeing a whole lot more of.  You can easily  find the book “CUDA by Example” It requires basic to intermed iate C skills, but not OpenGL or other graphics languages at all – you learn CUDA C as you go and start writing parallel code very quickly.(Amazingly, the run time just integrates the GPU device code and host code, and you can invoke code to run on the GPU as easily as any C function. It is some hot stuff behind the scenes, but surprisingly easy to code.)