Environmental Scenario Search Engine

Web services for the ESSE authoritative data sources and fuzzy search engine, as well as the prototype user interface will be installed on two mirroring servers one in the U.S. and one in Russia.

Authoritative data sources

The real connection between the ESSE system and a given user community are the data sources that support it. It will be relatively easy to add a new data source to the ESSE through the web-services interface, so the list below should not be taken as limiting but rather as a starting point that demonstrates the ESSE functionality.

Data Source	Type	Sample Parameters	Temporal Coverage	Spatial Coverage	Size	URL
NCEP/NCAR	Meteo.	Temperature, Windspeed, Cloud Cover	1949 - present	Global @2.5 Deg.	250 Gb	http://dss.ucar.edu/pub/reanalysis/
SPIDR	Space	Kp index, Sunspot Number	1933 - present	Global	30 Gb	http://spidr.ngdc.noaa.gov

The first thing to notice is the relatively large size of the archives. Using the distributed database concept allows us to perform interactive mining on these substantial data sources. The second thing to notice is the long temporal ranges. The ESSE will be most useful when the size of the archive prohibits or makes impractical searching by hand.

To describe each resource briefly, the NCEP/NCAR reanalysis data archive was derived from numerical weather prediction model runs. It represents gridded output on a regular time step (6 hours) and fixed grid step (2.5 deg). The model uses data ingest procedures to assimilate observational data into model results to produce a consistent picture of the terrestrial weather during the last 50 years. The Space Physics Interactive Data Resource (SPIDR) is an observational data source which incluces the output of numerical models. The SPIDR system currently handles the following: Defense Meteorological Satellite Program (DMSP) visible, infrared and microwave browse imagery, ionospheric parameters, geomagnetic variations, geophysical and solar indices, GOES satellite x-ray, plasma, and magnetometer data, cosmic rays, and solar radio telescope data sets.

The ESSE would also plan to add gridded space weather, ocean and terrain data in the near future, making the ESSE mining technology available across a wide representation of the "Digital Earth" environment. To add a new data source to the ESSE it will require: a) to write a web-service implementing standard interface with 2 methods, getMetadata and getData, which subsets the data source into a simple data model; b) create metadata document following pre-defined XML schema describing parameters, time and spatial coverage of the data source; c) add the web service address and credentials to the ESSE configuration.

Fuzzy search algorithm

People often use qualitative notions to describe such variables as temperature, pressure, pulse rate. In reality, it is difficult to put a single threshold between what is called "warm" and "hot". Fuzzy set theory serves as a translator from vague linguistic terms into strict mathematical objects. This is exactly what is needed to bridge the gap between current environmental archives and the policy makers, users and scientists who need to access them.

Intelligent environmental scenario searching across the distributed resources will be performed within the ESSE fuzzy search engine. The scenario editor from the ESSE user interface will be used to formulate a set of conditions to be satisfied by the candidate events. The search conditions may be specified in a number of ways depending on the user's familiarity with the region/data of interest. An expert user can specify exact thresholds and/or limitations that must be maintained on certain parameters. Conditions can also be specified via abstract natural language definitions for each parameter. For instance, temperature limitations can be specified as "hot", "cold", or "typical". The query can also be specified in terms of predefined rules which collect conditions into a named set. Thus, a user can specify the following weather search request:

(VERY LARGE "precipitation rate") AND ("surface temperature" ABOUT 10 C) AND (LOW "vertical wind speed at pressure level 1000 mbar")

or the following search request for severe magnetic storms in the space environment:

(VERY LARGE "Kp index") AND (VERY LOW "Dst index").

The result of such a request reported by the fuzzy search engine will be a list of the "most likely" dates for the event ranked by the sorted values of the aggregated multidimensional fuzzy membership function (MF). The aggregation will be done using fuzzy analog of the logical AND operator.

The ESSE client application will be searching for events in the environment where the input variables and the one-dimensional MFs depend on time, as well as the fuzzy AND aggregation of the desired conditions. We consider the values of the resulting time series as the "likeliness" that the environmental event to occurred at the time moment t, and search for the highest values of the aggregated MF and consider these to be the most likely candidates for the environmental events.

To be able to search for events like "the hottest day" or "the hottest week" we introduce the concept of event duration. For example, the time step of the parameter from the NCEP/NCAR reanalysis database is 6 hours, so the minimum event duration is also 6 hours, but the event duration could be also 1 day, 1 week, etc. We will do a moving average of the input parameters with the time window of the event duration before calculation of the one-dimensional MFs and the fuzzy AND aggregation.

Possible use

The applications of the ESSE systems are broad. As more and more data archives become available through projects like CLASS (NOAA), EOSDIS (NASA), DODS (Univ. RI) and other network accessible data systems, the tools to extract information from them become more valuable. As Nature declared in a 1999 article (Reichhardt 1999) "It's sink or swim as a tidal wave of data approaches". ESSE can help users prepare by providing tools which sift through the vast quantities of data available on-line and point at the interesting bits. This means that even with the volume of data increasing so rapidly and the number of researchers remaining relatively level we can hope to extract the most valuable information from the observations and carry that back to the relevant scientific communities.

The application of fuzzy logic based data tools goes far beyond simple event selection. For example an ever present issue when dealing with these large data sets is quality control. There is simply too large a volume to reasonably screen by hand. Searching capabilities can be used, for example, to analyze climate trends. Using techniques such as peer-matching and expert systems we can extend the ESSE to monitor data and alert data managers to changes and anomalies. As the computational power available expands we can extend the system into areas such as data classification whereby we can identify modes of the environment and perhaps identify new unknown relations in specific regions.

Finally the emergence of a network infra-structure for data access is providing new opportunities for the scientific researcher. It is now fairly trivial to reach out across discipline boundaries and access data in an immediately useable format. This is true for example in the case of the terrestrial weather community being able to make use of the space data made available through the SPIDR, for example, to study the influence of space weather on the Earth's climate. With these opportunities come challenges. As researchers expand into domains in which they may not be expert they will come to rely on intelligent tools to support them.

The mission of the ESSE is fundamentally to help a user distill the vast amount of available data down to a manageable amount of information. Beyond this however the ESSE has applications in the area of data quality control, data classification and even forecasting. The increasing data volumes available in the future demand different techniques to handle it and the ESSE framework is one exceptional method for a user to handle it.

The found event can be used as a source of a real scenario for computer games and simulators. Project deliverables

1. Web-service for fuzzy logic search engine
2. Web-services for several authoritative environmental data sources
3. Prototype web-application
4. Documentation including user guide, programmer guide, and the system whitepaper
5. Web links for free download of the software sources and the environmental databases

In order to enable true platform independence the team is working in both Open Source and Microsoft ASP.NET web services infrastructures, developing scientific software with the same external and internal interfaces. The same portal user interface can consume either of implementations.

Microsoft .NET Framework includes comprehensive set of classes that supersedes many of the commonly used open source libraries. This is especially true for XML processing libraries and enabled easy creation of wrapper classes ensuring portability of the source code created in this project.

References

ESSE Enabling Technologies