Scope

TNPs should check all files for data quality before submission and verify that they pass at least the following tests. Additional tests as each TNP sees fit are highly encouraged, as is reaching out to the City of Chicago for any discussion each TNP thinks useful in this effort.

Errors detected should be fixed before file submission. If it is impossible or unreasonably difficult to fix the errors, TNPs should inform BACP of the nature and degree (such as number of records affected) of the known errors in a file.

This page does not yet cover the most recent changes to the other pages in this reporting manual. TNPs should follow the spirit of the specific items below – checking all data for accuracy and consistency – in reporting new or changed columns/files.

File Format

Standard #	File(s)	Standard	Test
1	Driver	Driver ID columns complete	D1, D2, and D3 contain no blanks.
2	Session	Driver ID columns complete	S1 and S2 contain no blanks.
3	Trip	Driver ID columns complete	T1 and T2 contain no blanks.
4	Vehicle	Driver ID columns complete	V8 and V9 contain no blanks.
5	Trip	Shared trip columns complete	T14 and T15 contain no blanks.
6	Vehicle	Vehicle ID columns complete	V1, V2, and V3 contain no blanks.
7	Trip	Vehicle ID columns complete	T3 contains no blanks.
8	Vehicle	Vehicle description columns complete	V4, V5, V6, and V7 contain no blanks.
9	Vehicle	WAV column complete	V10 contains either Y or N for every record.
10	Vehicle	Valid inspection date	V11 contains a date in the past for every record.
11	Driver	No duplicates	No combination of D2/D3 values appears more than once.
12	Vehicle	No duplicates	No V1 value or combination of V2/V3 values appears more than once, except if due to another driver (distinct V8/V9) also using the vehicle.
13	Driver Session	Session and Driver files match	All S1/S2 values have corresponding D2/D3 values.
14	Driver Trip	Trip and Driver files match	All T1/T2 values have corresponding D2/D3 values.
15	Driver Vehicle	Vehicle and Driver files match	All V8/V9 values have corresponding D2/D3 values.
16	Trip Vehicle	Trip and Vehicle files match	All T3 values have a corresponding V1 value.
17	All	Valid data types	All non-string columns contain values valid for the column type.
18	All	Valid coded values	All columns that permit only a defined set of values contain only those values

File Content

The above tests are designed to detect structural errors in the data files. TNPs should also make reasonable efforts to check for situations in which data are structurally valid but incorrect. While no list can be comprehensive, examples of things to check might include:

• The number of records in the file is correct.
• No records have been excluded in any reasonably detectible way.
• No excess records have been included in any reasonably detectible way.
• All values fall into plausible ranges.
• Totals, averages, and other summary measures are consistent with internal benchmarks.
• Continuous improvement: If a problem occurs in one period – and especially in multiple periods – add it to the test procedures for future periods. It is understandable if it never occurred to a TNP that a particular error could occur. However, after it does occur and is detected, it is reasonable to assume it might happen again and should be detected and corrected before file submission.