March 01, 2015 / by Open Data Portal Team / In Open Data , Data Portal

Dataset updates are working inconsistently - 2/28/2015

Many of the automated jobs we use to update datasets were taking much longer than usual to run or not completing at all today, although we did not discover the problem until this evening.

Because all of last night’s overnight jobs completed successfully as far as we know, most datasets should be up-to-date and we think tonight’s overnight jobs should complete, although possibly not by their normal times.  Because it is a weekend and Monday is a City of Chicago holiday (Pulaski Day), most source systems should have little change until Tuesday (appearing on the portal early Wednesday, in most cases), anyway. 

We have temporarily reduced the frequency of our every-15-minute updates (Towed Vehicles, Relocated Vehicles, Traffic Congestion by Region, and Traffic Congestion by Segment) to once an hour and Public Chauffeurs from once an hour through most of the day to once a day at 11:30 pm.

We are working with our Data Portal vendor and will post any significant updates, either as modifications to this post or new posts.

We apologize for the inconvenience.  If any of the schedule changes would create problems for your processes, please contact @ChicagoCDO and/or @JonOpenData on Twitter and we may be able to make adjustments to meet your needs.

Update 3/3/2015, 2:50 pm - We restored all updates to their normal frequencies about three hours ago.  A few of the jobs since that time have not succeeded but, for the most part, we appear to be back to normal operations.  We are continuing to monitor but there will be no further updates to this post unless significant problems recur.

February 27, 2015 / by Open Data Portal Team / In Open Data , Data Portal

Dataset updates are failing - 2/27/2015

Many of the automated jobs we use to update datasets are failing. This problem started at about 9 am so the bulk of our datasets that update overnight are current.  The only known exception is Speed Camera Violations, which coincidentally failed last night and cannot be rerun manually at this time.

The main effect is on our datasets which update every 15 minutes (Towed Vehicles, Relocated Vehicles, Traffic Congestion by Region, and Traffic Congestion by Segment) and every hour during business hours (Public Chauffeurs).

We are working with our Data Portal vendor and will post any significant updates, either as modifications to this post or new posts.

We apologize for the inconvenience.

Update 5:20 pm - The updates are working again.  Every dataset mentioned above, including Speed Camera Violations, has gone through at least one update since 5 pm today and appears to be updating on its normal cycle.

February 26, 2015 / by Open Data Portal Team / In Open Data , Data Portal

Removing Bike Route map datasets in outdated formats

We have removed the shapefile (2wak-k8cp) and KML (kmdg-2wwn) versions of the current Bike Route dataset, which is in our newer, interactive format.  This is just a housekeeping action since the new dataset has been on the Data Portal for quite a while, contains the same version of the map, and allows for exporting both the shapefile and KML formats.

February 12, 2015 / by Open Data Portal Team / In Open Data , Data Portal

Added four years of records to Business Licenses

We have expanded the beginning Date Issued in the Business Licenses dataset to 1/1/2002.  It previously began at 1/1/2006.

As part of this change, we also upgraded the process that updates the dataset daily.  All indications are that the new process is working properly but please let us know if you find any issues.

December 03, 2014 / by Open Data Portal Team / In Open Data , Data Portal

Column type change in Relocated Vehicles

We have changed the “Relocated To Address Number” column (API Name: relocated_to_address_number) in the Relocated Vehicles dataset from Number type to Plain Text.

Even though the column usually should contain the numeric portion of the address, occasional errors in the source system result in text values that would interrupt our new update process.  The corresponding “Relocated From Address Number” (API Name: relocated_from_address_number) has been Plain Text for quite a while.

October 31, 2014 / by Open Data Portal Team / In Open Data , Data Portal

Dataset updates are failing - 10/31/2014

Many of the automated jobs we use to update datasets are failing. This problem started at about 9 pm last night and affected our overnight jobs that feed many of our regularly updated datasets.

We are working on the problem, as well as retrying individual jobs as they fail — although they often fail again.

In almost all cases, our jobs are designed to catch up if previous runs have failed.  We are not aware of any failures today for the small number of datasets that would not be able to make up for missed runs.  Therefore, we are not expecting any data to be lost permanently.  Datasets just may not be as up-to-date as usual until this problem has been resolved.

We will post any significant updates, either as modifications to this post or new posts.

The tags below include the database identifiers (often known as a 4x4 or four-by-four) for datasets known to be affected, although this might not be a complete list.

We apologize for the inconvenience.

Update 12:45 pm - The updates are working again. Some are still in progress so the datasets do not yet show new data.

October 22, 2014 / by Open Data Portal Team / In Open Data , Data Portal

Socrata's API Changelog

If you are using the API to extract data from our portal, another good site to follow is Socrata’s API Changelog.

October 08, 2014 / by Open Data Portal Team / In Open Data , Data Portal

How to load a dataset from the 4x4

As mentioned in the previous post, every dataset on the Chicago Data Portal has a unique identifier, often known as a 4x4.  It is the string of four alphanumeric characters, a dash, and four more alphanumeric characters found at the end of the URL.  For example, in https://data.cityofchicago.org/Public-Safety/Crimes-2001-to-present/ijzp-q8t2, it is ijzp-q8t2.

We typically tag posts on this blog with the 4x4’s of the affected datasets.  You can quickly load the corresponding dataset at the URL https://data.cityofchicago.org/id/[4x4] – for example, https://data.cityofchicago.org/id/ijzp-q8t2.

October 08, 2014 / by Open Data Portal Team / In Open Data , Data Portal

Dataset updates are failing - 10/8/2014

Many of the automated jobs we use to update datasets are failing. This problem started to a limited degree yesterday afternoon, causing us to miss some updates on datasets that update frequently during the day, but became more severe with our overnight jobs that feed most of our regularly updated datasets.

We are working on the problem, as well as retrying individual jobs as they fail – although they often fail again.

In almost all cases, our jobs are designed to catch up if previous runs have failed.  We are not aware of any failures today for the small number of datasets that would not be able to make up for missed runs.  Therefore, we are not expecting any data to be lost permanently.  Datasets just may not be as up-to-date as usual until this problem has been resolved.

We will post any significant updates, either as modifications to this post or new posts.

The tags below include the database identifiers (often known as a 4x4 or four-by-four) for datasets known to be affected, although this might not be a complete list.

We apologize for the inconvenience.

Update 10/9/2014 - The automated jobs have been working again since yesterday afternoon.  We are cautiously optimistic that the problem has been fixed.

October 02, 2014 / by Open Data Portal Team / In Open Data , Data Portal

Updated Zoning map and format

We have created a new Boundaries - Zoning Districts dataset in our newer, interactive format.  See our 8/11/2014 Deprecating several attachment datasets on August 31 post for discussion of this new format.

We have deprecated the previous, attachment-only Shapefile and KML datasets.