Algorithm for Predicting Useful Information DevelopedCategory: Science & Technology
Posted: July 29, 2014 06:13AM
Information can be very overwhelming for humans, which is part of the reason computers were developed, but even they can struggle. Massive datasets can slow even the best computers, and when results of some analysis are needed urgently, that can be a problem. Researchers at MIT though have developed an algorithm that can intelligently predict what information in a dataset will be useful, without the slow task of directly analyzing it.
To achieve this, the researchers turned to a probabilistic graphical model, which abstracts data into nodes, with connecting edges representing relationships between the nodes. By knowing the strength of the connections, one can quickly target what data is most valuable, and focus on them. Determining the strength can be complicated if nodes are connected by more than one path, creating a loop. To address this, the algorithm creates a spanning tree that dispenses with the loops and turns to Gaussian distributions to avoid distortion. It turns out that the probabilities represented by the graph are Gaussian, which means they can be described by their average value and variance. The uncertainty of the problem can be determined from the variance, but that does not require actually processing the data.
What all of this adds up to is a way to identify the most important and useful information in a dataset, without having to analyze the dataset. This results in the process being significantly faster, which could be very important if, for example, it is weather data being used to predict a storm's path that is being analyzed.