Data quality tool - Revision history

Lensovet: note why we're not doing travel-time estimates

2010-11-19T01:46:47Z

note why we're not doing travel-time estimates

Lensovet: updates reflecting discussions with seb and samitha

2010-11-18T23:08:37Z

updates reflecting discussions with seb and samitha

Lensovet: address Seb's comments

2010-11-14T06:45:38Z

address Seb's comments

Lensovet: Created page with 'With a commoditization of hardware and associated decreasing infrastructure costs, the number of ways in which the data needed to drive traffic monitoring has increased significa…'

2010-11-13T00:37:49Z

Created page with 'With a commoditization of hardware and associated decreasing infrastructure costs, the number of ways in which the data needed to drive traffic monitoring has increased significa…'

New page

With a commoditization of hardware and associated decreasing infrastructure costs, the number of ways in which the data needed to drive traffic monitoring has increased significantly. However, most traffic measurement methods have drawbacks related to the technology which they use. For example, data collection via smartphones has extremely low infrastructure costs and the ability to provide extremely precise and granular data but suffer from extremely low penetration. Inductive loop detectors, on the other hand, suffer from being point-based measuring tools which cannot always be used for time travel estimates, can carry high maintenance costs, and are not always accurate; at the same time, they have extremely high penetration rates (in locations where they are installed).

This diversity and difference in strengths and weaknesses makes it evident that it is necessary, at least at this point in time, to be able to combine different data sources together into a single "traffic picture" whose data is of higher quality than that of any individual source. A key component of such a system is the ability to evaluate the quality of both the individual data streams as well as their combinations to determine what improvements, if any, exist in the new, combined system, as well as whether certain parameters or combination provide better results. The goal of this project is to devise such a data quality assessment system so that these evaluations and analyses can be made.

==General system description==
The system, as currently envisioned, will be a web-based portal which will allow users to evaluate the quality of various data feeds through any modern, standards-compliant browser with an internet connection to CCIT servers. The interface will be primarily visual, allowing users to compare a number of metrics (described in more detail below) visually as well as numerically. At the current time, a comparison of at least one and at most two data sources will be possible, though the system will be designed in such a way that the latter restriction will not be permanent and could be lifted in the future.

In the spirit of modern and user-friendly web design paradigms, the system should be responsive and visually appealing. Exporting data for sharing via email or other means should be easy and not painful. The tool should be useful "as is" or "out of the box" but still allow a useful amount of customization to allow users to tailor it to their specific needs or application.

Note that in general, the CCIT system contains two types of data feeds: travel time distributions (such as the FasTrack travel time system) and point-based speed/flow/density estimates. In the short term, the system will be designed with the latter group of feeds in mind. However, it should be architected in such a way that the addition of travel time feeds should be possible without much additional effort.

==Evaluation metrics==
The above-described system is largely useless without a description of the metrics which it will display for analysis. It is implied, of course, that this list of metrics is not all-inclusive, and the system should be designed in such a fashion that the addition of new metrics should be trivial beyond the coding of the metric calculation itself. As per the earlier mention that point-based-data evaluations are the first priority, the metrics below are only applicable to such sources.

The data quality assessment tool should provide an easy-to-use interface to specify "correct" or benchmark values for each of the metrics, as the tolerable amount of, for example, GPS error depends on the specific application for which the data is being considered. Also, any feed available to the system should be usable as a benchmark for any metric (as long as it has the data to calculate it).

===Data-level metrics===
*Distribution of GPS errors (as reported by the recording device)
*Distribution of map-matching errors (as determined by MM mapmatching algorithms)
*Density of data (total number of points per link)
*Frequency of new data receipt (total per link)
*Data transmission delay (time difference between data recording and data storage on server; 2-step delay for TeleNav only: device→TeleNav server→CCIT server)
*Distribution of point location distance from link end (for city locations with traffic lights, provides ability to flag when most data points are not on the ends, since people should be waiting at lights)

All of these metrics should be filterable by time intervals, feed, device model (if available), location (specified as a network, polygon, or set of specific links), and unique device. This would allow for the analysis of derived metrics such as "density of data per unique device" or "distribution of point location from link end for the city of SF."

===Application-level metrics===
These metrics will allow the users to see the differences between feeds on the application level, i.e. when they are used as input to some model, rather than on the raw data alone. The addition of new metrics at this level is more constrained, as it requires that the application being benchmarked is (a) capable of accepting different feeds as inputs, and (b) is easily instrumented to calculate the metrics of interest. Two main metrics are envisioned for the system at this point:

*Value of information from adding an individual feed into a model
*Value of information from adding multiple feeds into a model

The value of information is defined as the degree to which the distribution of travel times from some model more closely resembles some "ground truth" distribution previously obtained. The difference between adding a single feed and multiple feeds is fairly self-explanatory: the former evaluates the value of information from adding a single feed to some baseline model, whereas the latter does the same but for more than one feed. Note that these are different metrics, because the improvement in the model output neither linear nor monotonically additive in the positive direction. As a result, certain feed combinations could actually worsen model performance, while other feeds which provide dramatic improvements individually may not add much when other feeds are added at the same time.

@@ Line 74: / Line 74: @@
 For comparing travel time feeds, the system should provide metrics which are applicable to distributions of values instead of individual values. For example,
 *The Wasserstein metric (comparison of distributions) (estimated travel time distribution vs. collected samples)
 ===Link-based feeds===

@@ Line 6: / Line 6: @@
 The system, as currently envisioned, will be a web-based portal which will allow users to evaluate the quality of various data feeds through any modern, standards-compliant browser with an internet connection to CCIT servers. The interface will be primarily visual, allowing users to compare a number of metrics (described in more detail below) visually as well as numerically. At the current time, a comparison of at least one and at most two data sources will be possible, though the system will be designed in such a way that the latter restriction will not be permanent and could be lifted in the future.
-In the spirit of modern and user-friendly web design paradigms, the system should be responsive and visually appealing. Exporting data (both in tabular and graphical format) for sharing via email or other means should be easy and not painful. The tool should be useful "as is" or "out of the box" but still allow a useful amount of customization to allow users to tailor it to their specific needs or application.
+In the spirit of modern and user-friendly web design paradigms, the system should be responsive and visually appealing. Exporting generated graphs for sharing via email or other means should be easy and not painful. The tool should be useful "as is" or "out of the box" but still allow a useful amount of customization to allow users to tailor it to their specific needs or application.
 More details on the user interface and program layout will be specified in a separate document. The system will be targeted primarily at CCIT researchers familiar with the different data feeds and models which are available. However, the user interface will be designed with less technical users in mind, so that (for example) it can be demoed live to transportation professionals.
-Note that in general, the CCIT system contains two types of data feeds: travel time distributions (such as the FasTrack travel time system) and point-based speed/flow/density estimates. In the short term, the system will be designed with the latter group of feeds in mind. However, it should be architected in such a way that the addition of travel time feeds should be possible without much additional effort.
+Note that in general, the CCIT system contains two types of data feeds: link-based and point-based. In the short term, the system will be designed with the latter group of feeds in mind. However, it should be architected in such a way that the addition of link-based feeds should be possible without much additional effort.
 ==Evaluation metrics==
@@ Line 19: / Line 43: @@
 All metrics, whether direct or calculated, will generally be generated on-the-fly for each request. As such, the system does not require the use of a database for storing calculated data. If system usage is high enough that database load or performance become a problem, we should look into using [http://www.memcached.org memcached] for storing the calculation results. Since the results are transient, storing them permanently in a database does not make much sense.
-It is possible that the system will allow users to flag specific, problematic data points or feeds. Since this feature adds the overhead of having to store this data in a manageable and useful fashion, it is not clear if this will be implemented in the initial release.
+The system will not allow users to flag specific, problematic data points or feeds initially due to the added overhead of having to store this data in a manageable and useful fashion. If users want to, in effect, create a new output filter based on their analyses, the tool will provide a textual summary of the user's current filtering algorithm so that a corresponding output filter may be created by the system administrator.
 ===Data-level metrics===
@@ Line 25: / Line 49: @@
 *Distribution of map-matching errors (as determined by MM mapmatching algorithms)
 *Data transmission delay (time difference between data recording and data storage on server; 2-step delay for TeleNav only: device→TeleNav server→CCIT server)
 All of these metrics should be filterable by time intervals, feed, device model (if available), location (specified as a network, polygon, or set of specific links), and unique device. This would allow for the analysis of derived metrics such as "density of data per unique device" or "distribution of point location from link end for the city of SF."
-===Application-level metrics===
+===Application-level comparison===
-These metrics will allow the users to see the differences between feeds on the application level, i.e. when they are used as input to some model, rather than on the raw data alone. The addition of new metrics at this level is more constrained, as it requires that the application being benchmarked is (a) capable of accepting different feeds as inputs, and (b) is easily instrumented to calculate the metrics of interest. Two main metrics are envisioned for the system at this point:
+This functionality will allow the users to directly compare the output of some model when different combinations of input feeds are used with it. At this time, the comparison between outputs will be purely visual, though specific metrics or analytical/numerical comparison methodologies can be added at a later time.
-The value of information is defined as the degree to which the distribution of travel times from some model more closely resembles some "ground truth" distribution previously obtained. The difference between adding a single feed and multiple feeds is fairly self-explanatory: the former evaluates the value of information from adding a single feed to some baseline model, whereas the latter does the same but for more than one feed. Note that these are different metrics, because the improvement in the model output neither linear nor monotonically additive in the positive direction. As a result, certain feed combinations could actually worsen model performance, while other feeds which provide dramatic improvements individually may not add much when other feeds are added at the same time.
+The system will be designed as follows: for a chosen model (i.e. the highway model), the user will be able specify which input feeds should be used and for what time period the model will be run. The user will provide a contact email address as well. When the user submits the request, their task will be placed into a queue of model runs. When the model finishes computation, the user will be sent an email containing a link at which they may view the model's results. These results will be viewable to other users as well. As a result, if the user wants to compare the output of the highway model on two different sets of input feeds, he may either:
 ==Future directions==