Our review process for OSINT operations
Our review process exists to give the best possible scrutiny framework to the reader and provide context and information about the steps taken to ensure the neutrality and accuracy of our findings. One can see it as a bunch of metadata about the investigation, with an appreciation of how well we followed the rules we set for ourselves. All this in an effort to achieve transparency and to ensure our readers that all our investigations follow the same, replicable standards.
Check First helped to design and follow the guidelines of the EFCSN for OSINT operations. Using the words of the authors of the guidelines: “In recent years, online investigations using openly available data has flourished. Inspired by the work of Bellingcat and other global leaders, OSINT has raised in many online conversations. Sometimes, it has also been used as a buzzword covering all type of online investigation. With an OSINT community being built, and the lack of common standards, multiple OSINT practitioners felt it was time to put together conceptions, methodologies, and good practices of their OSINT work. For practical reasons, they decided to focus their work on public-facing outcomes of OSINT.”
Internal and external review
The process starts once an investigation paper draft is ready. Evaluation begins with a double assessment by two distinct internal auditors from the Check First staff. They will go through a checklist of 18 items (see here) and annotate it. If all items are checked, the assessment score will be 100% and the score will decrease as less items are checked.
Four item categories are examined for the assessment of any of our OSINT operations:
- Quality Assessment
- Accuracy
- Sampling and Bias
- Attribution
Once this internal review step is done, the draft is sent to two different contracted external auditors for a first assessment. These auditors are chosen based on their qualification on the topic of the investigation. A contract is signed each time in between Check First and each external expert, including an independence and impartiality clause. This clause guarantees that the experts will assess the paper regardless of any possible compensation from Check First for the reviewers’ time.
After a first external review, Check First may modify the draft to take into account remarks from the external auditors. If this is the case, a second and final assessment is performed by the external auditor who raised issues, or by both if both raised issues.
Publication
Once all reviews are done and the paper is ready to be published, the final assessment score of the paper is calculated from the average of all reviews. This score is intended to communicate to the reader how well our standards were observed. For any OSINT investigation to be published, this score must be above 80%.
The readers can also get information about the process and its auditors as it is required that the qualities of internal and external reviewers are included in the published paper, as well as the final assessment score and a link to the assessment review template document. Reviewers sign the exported pdf. The final document includes a version control table showing version numbers increased upon each modification and an annotation of the changes made.
Our hope is to communicate to the best of our ability the trustworthiness of our work and to prevent partiality or bias.