KSH statistics and raw datasets – import and storage

The indicator import, the file download, CSV and SDMX-ML processing, storage, the chart endpoints and the permission keys.

Overview

ksh/stat imports and charts NUTS3-level indicators, ksh/dataset browses the raw rows of downloaded datasets. Both resolve the active API configuration with code "ksh" (company → group → global).

Data model

Table names carry the installation's own prefix in the database.

Table or location	Role	Key, main fields
`ksh_settlement_stat`	indicator values	unique key (geo, year, indicator), value
`ksh_dataset_row`	raw data rows	dataset, resource, geo, period, indicator, unit, value, `row_json`
`DIR_STORAGE/tmp_ksh/`	temporary download folder	CSV and XML files

Process

stat.import: per the stat_indicators list (empty: the 2 built-in indicators) downloads each resource once (ksh->datasetData), filters to NUTS3, applies an optional column filter, normalises the number and upserts; on partial failure the errors are reported too.
stat.fetchfiles: downloads the datasets of raw_datasets (or one dataset parameter) into tmp_ksh/ (ksh->downloadDatasets).
stat.processfiles: info=1 lists the downloaded CSVs; file= batch-upserts one CSV into the raw table. The file name passes basename, only .csv; the file is not deleted here.
dataset.sync: info=1 returns the (dataset, resource) pairs to process; for one resource: download, delete earlier rows (deleteByResource), process CSV or SDMX-ML XML with batched upserts, then delete the file. ext is only csv or xml.
Charts: stat.chartdata in county mode (latest year per county) or trend mode (time series of one county); dataset.chartdata in auto mode.

Permissions

ksh/stat: list (list and chart), import, delete, debug.
ksh/dataset: list, with a content scope on geo codes (ksh_dataset / geo rules); import, delete, debug. dataset.clear (full truncation) is super admin only.

Background job

The direct entry point cron/kshfetch does the same download and processing as "Fetch & process"; it is not part of the central job queue, and no authentication check is visible in the controller.

Known limitations

The dataset.chartdata endpoint does not apply the geo-code content scope.
After processfiles the CSV stays in the download folder.

Related topics: KSH data statistics background job

Request consultation Back to home