- A Multidisciplinary Wide Area Sensor Network
Interfacing the World with a Distributed Supercomputer
Prof. Dr. Heino Falcke
The world breathes, changes, and lives and we want to observe, understand and shape this world. Understanding the dynamic world around and above requires us to obtain, process, and digest an enormous amount of information. This can only by achieved through modern information and communication technology: it requires the construction of Wide Area Sensor Networks (WASN). A WASN consists of an array of geographically distributed sensors which are connected by a self-organizing, partly wireless network and an ultra-high speed backbone to a distributed super computer. All three components are mutually dependent: network and computing resources are constantly fed by a multitude of sensors providing a huge data flow that in turn could only be processed by this special high-throughput networks and computers of the next generation. LOFAR is the first large scale attempt to implement such a concept in the real world on a large scale, driven by a wide range of user knowledge processing user groups from a variety of disciplines. This includes but is not limited to astronomy, geophysics, meteorology, and agriculture where it will be a major player. LOFAR will open up the next level of distributed computing and networking and provide significant visibility for the technology partners involved.
Table of Contents:
The changing world around and above us provides an enormous amount of information that
can be measured through a distributed network of sensors, providing a continuous data stream
needs to be channeled to central data bases and monitoring and control units
needs to be processed for our information society to understand, predict, and shape our environment.
How can one connect different sets of geographically spread-out sensor arrays to a super computer?
A wide Area Sensor Network needs three basic components:
a large number of cheap and robust digital sensors of various kinds,
a dedicated high-speed (10Tb/s) network over several hundreds and thousands of kilometers,
very high performance central and distributed super computing capabilities.
Each of these areas requires new technologies and research and can form the basis of a next level of information services for science as well as for governments and citizens.
With the huge amount of information being generated by a large set of continuously streaming sensors, a WASN has demands beyond traditional high-bandwidth networks and applications. Different user groups demand different sensors, bandwidths and modes of operations. Currently planned examples of sensors required by different user groups are:
radio dipoles for radio astronomy creating a virtual telescope, studying: cosmology, space weather, ionosphere, high-energy particles
particle detectors: measuring cosmic ray induced particle showers
geophones for monitoring tectonic activity and ground water levels through micro-earthquakes
weather stations for real-time weather and storm evolution monitoring
microphones for infrasound applications
soil, crop, and livestock sensors for enabling precision agriculture
The initial driver for LOFAR came from radio astronomy which has a tradition in digitally connected element interferometers and high-bandwidth applications. The idea for LOFAR was to replace a conventional radio telescope with a distributed network (phased array) of simple digitally sampled dipoles. This allows one to realize a telescope completely in software which can point and focus electronically without any moving parts – in fact it can even look in multiple directions and trough data buffering even back in time. At low radio frequencies this is possible with modern technology and would provide up to three orders of magnitude increase in resolution and sensitivity over existing facilities. The idea of and desire for an astronomical Wide Area Sensor Network was born! Within an international consortium of scientific institutes the basic concept and requirements for LOFAR were specified:
2x 13,000 small antennas
clustered in 100 stations distributed over 350 km in a “log-spiral” configuration
>20 Tbit/sec data network (if full scientific bandwidth)
>40 Tflop distributed supercomputer
innovative software systems
data mining and visualization
full and exclusive control via the Internet
instantaneous view of the full sky, several simultaneous users.
Now, the LOFAR project is ready to make the next step…
In the course of the R&D for LOFAR the project realized that the genuine nature of their concept went beyond astronomy and fundamental science. As the technical concepts matured the interest from other scientific communities and from commercial parties steadily increased. The main aspects that make the LOFAR project attractive for other fields are:
With a dedicated high-speed network and 100 extended stations, additional sensors can be added at only a small additional cost.
Digital sensors are a main driver for high-speed networking and supercomputing, exceeding the demands and vision of multi-media applications or computer-to-computer networking.
A genuine WASN has never been realized on the scale of LOFAR.
The additional cost for this expansion of LOFAR is the burden of increased flexibility and a wide range of user demands in terms of bandwidth, robustness, interconnectivity, and services. For example, radio astronomy has a high demand on bandwidth and synchronization. Geophones, microphones, weather station have low-bandwidth but may require local pre-processing. Agricultural sensors can be widely distributed and non-stationary (e.g. cows!). High-energy particle detectors for astroparticle physics applications need special triggering and synchronization. All applications may also demand differently spaced network access points. As a result an interdisciplinary knowledge consortium is formed that clusters around a central ICT research and infrastructure, producing fundamental and applied research.
There are a number of key technologies that need to be developed by various partners in the consortium. Some of the key areas of research and development are
low-cost sensors: every user group has defined and design its own set of robust, affordable sensors that can be coupled to the network,
generic sensor interface: coupling sensors into the network requires the definition of a generic sensor interface that can accommodate the wide range of sensors envisaged,
broad-band data links: given the huge demand for bandwidth a (possibly routerless) optical network has to developed using a combination of existing dark fibers and newly laid connections,
IPv6 based protocols: a number of high- and low-bandwidth applications with guaranteed or ad-hoc allocated bandwidth have to communicate with sensors and operators and be integrated in a single network,
high performance cluster-computer: central processing requires not only fast number-crunching capabilities (10 Tflop) but also massive data throughput (25 Tbit/s) and buffering capabilities (up to several minutes).
embedded software is needed to implement the majority of the WASN functionality, especially the signal processing functions for the various sensor applications, and the handling of the dynamic and self-configuring behavior associated with some of these.
adaptive monitoring and autonomous control of the network has to cope with continuously changing environmental conditions, changing hardware configurations, hardware degradation, and changing user demands
intelligent web-based user interfaces: with the enormous amount of data being processed, the results have to be stored and then made accessible to a widely distributed user base via ‘traditional’ internet services, making the operation fully transparent, building up and providing access to a large interdisciplinary knowledge base.
While sensor development and data reduction algorithm will be the prime task of the various research communities and user groups, the remaining technology issues are well posed problems for commercial companies and public-private partnerships are the preferred model.
LOFAR is a unique project that combines some of the most advanced aspects of modern technology and science. As a major international science project it will provide large visibility for all partners involved through:
distribution: football field sized distribution of stations across the northern
excellence: LOFAR will be at the forefront of astronomical research which
nowadays guarantees media interest and public fascination for the next decade.
LOFAR is probably the only major astronomical facility that can still be realized
in the heart of
LOFAR will provide a strong link to students and schools across
Next generation networks: stress-testing IPv6 and security issues
Other new technology: The WASN concept will have many future applications beyond basic science that can affect everyday life and hence LOFAR will be a trend setter in this area.
Testbed: As a testbed LOFAR – driven by the high demands of basic science – allows one to develop and test already now techniques that markets will demand in the coming years.
There are numerous specific tasks for potential partners in the project. Some of those specific to the WASN testbed concept include, but are not limited to:
Lay out the LOFAR network on scales of 2, 50, 300 km distance with newly installed fibers (inner region) or using unmanaged dark fibers provided by a partner
Provide LOFAR nodes and stations with advanced GbE network equipment, GSM/GPRS links, GPS receivers, etc.
Provide various categories of sensors (e.g., measuring environmental parameters)
Develop protocols for simultaneous use of these sensors and secure communication protocols (IPv6 based - multitude of sensors, high performance secure transport of heterogeneous data, dynamic behavior through radio links or "ad hoc" behavior of some sensor types)
A wide area sensor network is a new and innovative approach to sensing the physical world. It is a technology that easily drives bandwidth demand to a new level, providing a step beyond GRID. The reason is that here we do not have computers talking to computers, but the real world talking to us! Computers and networks provide the key interface. The real world will always provide more information than we can handle in any virtual world, hence smart information reduction and processing is the key for a WASN. A European testbed, realized through LOFAR, driven by demanding users, will push this technology into a reality.
The early international
LOFAR consortium that initiated the project consists of ASTRON, MIT (Haystack
project has a strong regional backing in the northern Dutch provinces and on a
national level. Early 2003 a predominantly Dutch consortium has submitted a
70M€ proposal for funding of a significant part of LOFAR and a large program of
ICT related research that uses it. The consortium unites a number of Dutch
universities, research institutes, industrial enterprises, and some international
ASTRON has set up a professional management structure with a project office and system engineering team to handle the managerial challenges associated with this complex international situation. As of 2003, the total staff involved in LOFAR at ASTRON is 45.
ASTRON is a knowledge-institute for astronomical instrumentation funded by the Dutch state. The mission of ASTRON, the Netherlands Foundation for Research in Astronomy, is to “make astronomical discoveries happen by providing innovative observing instruments”. As such ASTRON is a technology provider for a very demanding customer base. We have a long standing experience in international projects, through our participation in international instrumentation development.
the Westerbork Radio Synthesis Telescope. Since 1993,
ASTRON has been working towards the next generation radio telescopes: the
Square Kilometre Array and LOFAR. The latter will be the first operational facility
that utilises massive signal processing, high‑speed data networks and
full internet based operations. ASTRON has a Technical Laboratory organised
around the competence areas of system engineering, antenna development, RF
electronics, digital signal processing, and software/image processing. The
Technical Laboratory also carries out major projects for optical, infrared and
millimetre instrumentation. ASTRON disseminates its knowledge and expertise to
regional SMEs through the SKAIHigh
program, sponsored by the
Prof. Harvey Butcher (firstname.lastname@example.org), ASTRON General Director
Dr. Mark Bentum (email@example.com), FP6 Consortium Coordinator
Prof. Heino Falcke (firstname.lastname@example.org), Senior Scientist LOFAR & European Coordinator
Dr. Marco de Vos (email@example.com), LOFAR System Engineering Manager
Tel: (+31) (0)521 595 100
Fax: (+31) (0)521 597 332