Skip to content

Parsing CSV-formatted gold-standard data

These instructions can be followed when the gold standard labeled data is in CSV format.

Expected input data

Each row in the CSV represents one document, with columns holding document metadata, bibliographic details, and human-labeled annotations. The header row must include at least the columns document_id, and name. The following columns would additionally provide an abstract, and an attribute column for an attribute named "mortality".

document_id,name,abstract,mortality

Column categories

Every column in the CSV falls into one of three categories.

Required metadata fields

document_id and name must always be present.

  • document_id — a unique identifier for the document.
  • name — the document's name. This can be anything, such as the paper title.

Reference fields

Columns that map to bibliographic metadata fields defined by destiny_sdk.enhancements. By default, a column is auto-assigned as a reference field whenever its name matches one of the allowed reference field keys below — for example, an abstract column is automatically treated as the document's abstract.

Allowed reference field keys:
  • abstract
  • authorship
  • cited_by_count
  • created_date
  • updated_date
  • publication_date
  • publication_year
  • publisher
  • title

Attribute fields

Any remaining columns are treated as gold standard annotations: labels that human annotators have assigned to each document (with disagreements resolved) to build a gold standard dataset. This dataset is later used to evaluate how well LLMs or ML models can classify or extract the same information.

In the example above, mortality is an attribute field. Its type is automatically inferred from the values in the column (allowed types are bool, int, float, and str). Here, mortality is inferred as a boolean, where True means annotators determined the study measures or analyzes mortality as an outcome, and False means it does not.

Key rule for the mortality example

A study must explicitly measure or analyze mortality as an outcome to be labeled TRUE. High lethality of the underlying condition alone is not sufficient.

Example data

document_id name abstract mortality
1 Mortality burden of diabetes attributable to high temperature and heatwave under climate change scenarios in China Climate change and diabetes pose the dual challenges to human health, yet there is a lack of evidence regarding future health burden of diabetes attributable to climate change. In this study, we used three-stage analytic strategy to project the heat-related and heatwave-related diabetes deaths by demographic characteristics and regions, during 2010-2100 in 32 major Chinese cities. True
2 Cholera forecast for Dhaka, Bangladesh, with the 2015-2016 El Nino: Lessons learned A substantial body of work supports a teleconnection between the El Nino-Southern Oscillation (ENSO) and cholera incidence in Bangladesh. In particular, high positive anomalies during the winter (Dec-Feb) in sea surface temperatures (SST) in the tropical Pacific have been shown to exacerbate the seasonal outbreak of cholera following the monsoons from August to November. Climate studies have indicated a role of regional precipitation over Bangladesh in mediating this long-distance effect. Motivated by this previous evidence, we took advantage of the strong 2015-2016 El Nino event to evaluate the predictability of cholera dynamics for the city in recent times based on two transmission models that incorporate SST anomalies and are fitted to the earlier surveillance records starting in 1995. False

Using this format to set up a project

1. Initialize the project

Once your CSV is ready, initialize a project and select the format generic_csv

deet project init

Or, in python:

from pathlib import Path

from deet.data_models.project import DeetProject
from deet.processors.converter_register import SupportedImportFormat

project = DeetProject(
    name="my project",
    gold_standard_data_path=Path("gold_standard.csv"),
    gold_standard_data_format=SupportedImportFormat.GENERIC_CSV,
)
project.setup()

See the quickstart guide for more on customizing prompts, linking PDFs, configuring extraction runs, and interpreting the results.