We're processing new FOIs — and shifting our methodology

The TLDR: We have started uploading the FOI request summaries for 2023/2024! Hooray! However in order to keep Secret Canada going — now that the team has been pulled onto new projects — we’ve had to make some changes. Primarily: We are going to focus on the hundreds of public institutions that are cooperating and cut ties with the bad actors who have been taking up the majority of our time for negligible benefits.

Let’s start with an apology: We are so sorry we have not been updating the news feed on Secret Canada. Our reporting jobs have moved us in new directions and it just hasn’t been feasible to be as on top of the blog as we had hoped. The reason is that — in addition to our new projects at The Globe — we have been busy filing new requests and collecting responses for our next release. 

And this is what we wanted to fill you in on today: We’re making some changes to Secret Canada’s methodology and process when it comes to the search database. (In case you’re new here: In 2023, we launched a tool that allows Canadians to search more than 300,000 previously completed freedom of information requests from more than 500 public institutions — city halls, government ministries and departments, police services, Crown corporations, universities, school boards, transit agencies, etc. — across the country. We collected this information through hundreds of individual requests to each public body.) We’re doing this so that we can realistically keep the project running, while also juggling our day jobs.

The major theme behind the updates is speed. We need the process to go faster. After three cycles of filing requests, we’ve become really good at that part. Where we need to streamline things is on intake, cleaning and posting.

First: we’re not going to release the 2024 data in one batch. We will be posting them as they come in — in fact, some have already been uploaded! Previously, we had waited until all the responses came in from everyone, then we cleaned everything, then we reviewed everything (we have to do this to try to prevent inadvertent sensitive details from making it online — for example, one time we spotted a child’s name and birthday left in an abuse report). From there, we uploaded everything in one giant batch. But this created a lot of bottlenecks. Going forward, there won’t be any big announcement. We’re updating all the time.

Second, we are going to address one of the major delay points. In the past, we have spent months chasing down public bodies that don’t respond to us, circling back three or four times before designating them as a “did not reply.” In future, we aren’t taking extreme efforts to collect data from delinquents. In fact, if a public entity has consistently shown us that they will not co-operate with The Globe’s requests, we are no longer going to FOI them. On one hand, this feels like giving up and letting the rule breakers succeed. On the other, for the greater good of the project, we’ve decided we would rather direct our limited resources toward efforts that will produce information for our users and readers. (Of note, this change means we are not going to FOI ministries in Alberta, as they have refused to comply with our requests. However, we will continue to watch for the outcome of this review.)

Next, we are going to eliminate responses that are not easily processed as an Excel sheet. When we file our FOIs, we ask public bodies to provide the information in Excel or .CSV format. The Secret Canada data intake tool requires us to upload the requests in this format. We explain this to the public bodies. Still, some have refused and have been sending us response letters as PDFs or printed-out copies sent through the mail. Sometimes, we’re able to quickly capture the words and convert these pages into an Excel document. But more often than not, the Secret Canada team has had to spend MONTHS cleaning the data — thank you, Mahima! — which sometimes involves manually typing out the request wording. Even some spreadsheets have caused problems, as the institution has used bizarre formatting that makes manipulating columns and rows nearly impossible. All of this is to say: Our new policy is that if we can’t fix the formatting in less than five minutes, we’re scrapping it. It’s just too much work. (For example, all provincial ministries in Saskatchewan will not be included going forward. The PDFs they send us are incredibly poor quality — sometimes illegible — and the province has refused to fix the error.)

Some other changes:

We are removing public institutions that do not have FOI tracking systems. Similarly, we are removing public institutions that have fewer than five responses. (Sometimes, when we’ve targeted smaller public bodies, the only FOI summary we got back was our own!) Again: we’ve decided to redirect efforts toward public bodies that can produce more useful information. We’re also removing institutions that continue to ask for — what we believe to be — excessive processing fees. The vast majority of public institutions provide the requested information at no charge as the data are readily available. Some have charged us a nominal fee, such as $10, $20 or $30. Those that continue to demand fees beyond this are being excluded. 

Finally, there are a few entities that have been removed because the information they provide is not helpful. For example, sometimes the “request summaries” are only one-word descriptors. Similarly, we are removing the Ministry of Environment in Quebec. This ministry’s response has consistently included more than 20,000 rows of request summaries, which are typically vague descriptors that provide minimal meaningful information. The resources required to review and clean this document are not worth it. 

In closing: We want to keep Secret Canada going! We want to continue to offer the largest database of completed FOI request summaries in the country from more than 500 institutions! To do this, we’ve had to be ruthless in assessing what’s possible with our time and resources.

Thank you for your continued support and happy FOIing!

We’d love to hear about how you’re using Secret Canada. Send us a note or use the hashtag #SecretCanada on social media. This information helps us grow the project.

To stay updated on FOI news, upcoming data releases and new features, sign up for The Globe’s Secret Canada newsletter.