HDF5 File Locking
Overview
TUFLOW FV users may wish to copy or review NetCDF and XMDF result files before a simulation has finished, for example to monitor model behaviour in QGIS, MATLAB or Python. In recent TUFLOW FV releases, HDF5 file locking can prevent these workflows. This page explains why file locking is used, the risks associated with disabling it and the recommended procedure for accessing result files while a simulation is running to minimise the risk of file corruption or incorrect file reads.
HDF5 is a data library used by TUFLOW FV to read and write NetCDF and XMDF files. TUFLOW FV 2026.0.0 to 2026.2.0 includes HDF5 1.14.6; TUFLOW FV 2025.2.2 used HDF5 1.8.16. File locking was introduced in HDF5 1.10 and is enabled by default in current HDF5 releases. It is intended to prevent file access patterns that could produce inconsistent reads or corrupt a file. However, file locking can restrict access to TUFLOW FV result files while a simulation is running. The exact behaviour differs between operating systems and file systems.
While TUFLOW FV is writing a NetCDF or XMDF result file:
- The file cannot normally be copied (Windows only).
- The TUFLOW Viewer QGIS Plugin's Copy HDF5 Files Before Loading option cannot successfully copy the file (Windows only).
- The file cannot normally be opened for review in QGIS, MATLAB, Python or similar software (Windows or Linux).
It is possible to override the default behaviour by setting the HDF5_USE_FILE_LOCKING environment variable to FALSE. The variable must be set in the environment of TUFLOW FV and every application that will access the file. Disabling file locking is an opt-in decision made at the user's own risk.
Warning: Disabling HDF5 file locking does not make live reading safe or coherent. It only removes the lock that would otherwise prevent the attempted access.
Risks Of Disabling File Locking
The principal risks are:
- Incoherent reads: A reader can silently receive inconsistent data while TUFLOW FV is writing. HDF5 metadata and raw data are not necessarily flushed in an order that provides a coherent view to a non-SWMR reader.
- File corruption: If more than one writer modifies the same HDF5 file at the same time, the file can be corrupted.
- Premature simulation termination: If another application opens the HDF5 file with locking enabled, including for reading, it can acquire a lock. On Windows, this may cause the TUFLOW FV process writing the file to terminate.
These risks are not eliminated by the procedure below. In particular, the risk of an incoherent read depends on the file system and remains present without Single-Writer/Multiple-Reader (SWMR) access.
If HDF5_USE_FILE_LOCKING is set to FALSE or BEST_EFFORT by default in the user environment, set it to TRUE before running TUFLOW FV unless file locking is being disabled deliberately for the workflow described on this page.
Disabling File Locking
The risks of file corruption and premature simulation termination can be minimised when TUFLOW FV and every reader follow the procedure below. The procedure does not guarantee that data read from an actively written file will be coherent.
1. Prevent Multiple Writers
Ensure that the same model is not being run simultaneously by multiple processes or users. TUFLOW FV will generally report an error if two simulations attempt to write the same result files, irrespective of the file-locking configuration; nevertheless, the output paths should be checked before starting a simulation.
2. Launch TUFLOW FV With File Locking Disabled
Set HDF5_USE_FILE_LOCKING=FALSE in a batch or shell script used only for the relevant TUFLOW FV simulation.
Windows Batch File
set exe="path_to_tuflowfv\TUFLOWFV.exe"
set OMP_NUM_THREADS=8
set HDF5_USE_FILE_LOCKING=FALSE
%exe% my_run_001.fvc
Linux shell script
#!/usr/bin/env bash
export OMP_NUM_THREADS=8
export HDF5_USE_FILE_LOCKING=FALSE
tuflowfv_2026.2 "my_run_001.fvc"
3. Launch Python, Jupyter Lab And MATLAB From A Configured Terminal
Python, Jupyter Lab and MATLAB must be launched from a terminal in which HDF5_USE_FILE_LOCKING=FALSE has already been set. This must occur before the application starts and before it loads the HDF5 library.
If a reader is started without the correct setting, TUFLOW FV may terminate with one of the following errors:
- An
nf90_createerror if the result file is already open when TUFLOW FV attempts to initialise it. - An
nf90_sync 101 NetCDF: HDF errorif the result file is opened while TUFLOW FV is writing it.
Important: For the HDF5 versions used by current TUFLOW FV releases, setting the environment variable from within an application after the HDF5 library has loaded may be too late.
For example, the following settings within a Python script, Jupyter Notebook or MATLAB session are not sufficient:
import os
os.environ["HDF5_USE_FILE_LOCKING"] = "FALSE"
setenv('HDF5_USE_FILE_LOCKING', 'FALSE');
Instead, set the variable first and then launch the application from the same terminal.
Windows Command Prompt
set "HDF5_USE_FILE_LOCKING=FALSE"
jupyter-lab
For MATLAB (or Python etc.), replace jupyter-lab with the command used to launch MATLAB (or Python etc.).
Windows PowerShell
$env:HDF5_USE_FILE_LOCKING = "FALSE"
jupyter-lab
Linux
export HDF5_USE_FILE_LOCKING=FALSE
jupyter-lab
For MATLAB (or Python etc.), replace jupyter-lab with the command used to launch MATLAB (or Python etc.).
4. Configure QGIS Before Opening Results
The QGIS process must also have HDF5_USE_FILE_LOCKING=FALSE set before it loads an actively written TUFLOW FV result file.
- In QGIS, open Settings > Options > System.
- Locate the Environment section.
- Add a custom environment variable named
HDF5_USE_FILE_LOCKINGwith the valueFALSEas shown in the image below. - Restart QGIS for the setting to take effect.
Provided TUFLOW FV was also launched with HDF5_USE_FILE_LOCKING=FALSE, QGIS should then be able to open the result file while the simulation is running. Any data displayed from the actively written file may still be incomplete or incoherent.
Alternatively, enable the TUFLOW Viewer's Copy HDF5 Files Before Loading option. For this option to copy a file on Windows, TUFLOW FV must have been launched with HDF5_USE_FILE_LOCKING=FALSE. A copied file is not guaranteed to represent a coherent snapshot if the copy was made while TUFLOW FV was writing it.
5. Do Not Disable Locking System-Wide
It is not recommended to set HDF5_USE_FILE_LOCKING=FALSE system wide. A system wide setting may affect unrelated applications that use HDF5 files. Set the variable only in the environment used to launch TUFLOW FV and the reader applications required for this workflow.
After completing the workflow, close the configured applications or restore the variable to TRUE before starting other HDF5 workflows.
Network File Systems
Additional caution is required when result files are located on network file systems, including NFS, SMB/CIFS, mapped network drives and UNC paths. File locking behaviour and write ordering guarantees can differ from those of local file systems. Where practical, run the model and review actively written results on a local disk, then copy completed results to network storage.
Future SWMR Support
HDF5 provides Single-Writer/Multiple-Reader (SWMR) functionality specifically for workflows in which one process writes a file while other processes read it. A future TUFLOW FV solution depends on suitable SWMR support being available through the publicly available NetCDF-C and NetCDF-Fortran libraries, followed by equivalent support in the TUFLOW FV XMDF libraries.
The TUFLOW Team intends to integrate this functionality when the required library support becomes available. Files written using SWMR may not be readable by software built against older HDF5 libraries. The network file system considerations above will also remain relevant because SWMR relies on appropriate write ordering guarantees.