Research Data Management
Documentation refers to any guidance that you are using to help control, manage or understand your collection of data.
This could be a set of guidelines, equipment set-up, description of fields on a dataset, specific mixes to create a paint colour or lab notebooks. These will not necessarily become part of your research output, but need to be managed in order to ensure the integrity of your output.
Reasons to document
- You may need to refer to documentation to understand your data in the future, should you need to refer to it or re-use the data
- So that collaborators and others can understand how the data is collected, so they can understand it, add to the data and analyse or re-use it.
- For review and publication, so you can talk about the data collection and analysis, for reproducibility and integrity purposes.
- To deposit the data into a repository, you may need to create certain documentation and answer certain questions. If you can find out about these early in the project, you can create the appropriate documentation as you go.
Metadata
Metadata is essentially 'data about data'. You give your data metadata to help others to discover it. For example, library records about an item in the collection are a form of metadata and they mean you can search through a library catalogue for titles, authors and other terms. If you deposit your data into a repository, you will be expected to provide metadata to enable discovery of your data.
You might follow a metadata standard relating to your chosen repository or another disciplinary standard. For example, you may need to use controlled vocabulary to describe your work, so that other searching using those terms can find your data. Standards can also be useful tools to aid you in describing your work adequately.
The 'Plan for Data Reuse' exercise from the Mozilla Science Lab may help you get used to describing your data appropriately for future re-use.
Ways to Document Data (Adapted from University of Bath, 2019)
Any information that cannot be recorded in a structured way (i.e. as the values of fields in a data or metadata file) can be recorded as free text within a readme file.
There are many commercial options here, but Jupyter Notebook is a good free option for researchers who are writing code or GitHub may be used to handle the project, versions and documentation. Evernote is a flexible tool and you can attach different types of files to notes, with a tagging system to help organise. MS OneNote is another free solution that integrates fully with Office 365 applications.
Some file formats can record information in addition to the main data content. For example, the Observations and Measurements XML standard provides a way of recording sampling strategies and procedures as well as measurement values.
Some disciplines have developed special file formats or data structures for recording supporting information. You can find more information on this below.
In some cases, archives/repositories generate specialist metadata files from their submission forms. Find out the fields of the submission form of the archive to which you are planning to submit your data, copy these fields into your data documentation and fill these in as you go through your project.
Some of the information needed to understand data would normally be provided in a journal article reporting the research. In order to prevent duplication of effort, it is possible to refer to an article to provide more information about a dataset, but before doing so you should be sure that (a) the article provides sufficient detail and (b) that the article will be available as open access.
Tips for Data Documentation (Adapted from University of Bath, 2019)
When documenting your data, the aim is to provide enough information so that a fellow researcher who is familiar with your field, but not necessarily with your work, should be able to understand the data, interpret it and use it in new research, without the need to contact you directly about the dataset.
An overview of the data should include:
- the methods used to collect the data, including major methodological decisions that have been taken;
- the structure of the files used;
- processing or data manipulation that has been undertaken to generate the results of the project.
Specifically, you may need to include some of the following information:
- details of the equipment used, such as make and model, settings and information on how it was calibrated;
- the text of questionnaires and interview templates or topic guides. If these are only available under licence details of how to access the instruments should be included;
- details of who collected the data and when;
- key features of the methodology, such as sampling technique, whether the experiment was blinded, how sample groups were subdivided;
- legal and ethical agreements relating to the data, such as consent forms, data licences, approval documents or COSHH forms;
- citations for any third-party data you have used;
- details of the file formats and standard data structures used to record the data and supporting information;
- a glossary of column names and abbreviations used, explaining for example which measurement resulted in a given column and what units were used;
- methods of managing missing data;
- the codebook you used to analyse and encode content;
- the workflow used to process and manipulate the data, including steps such as applying statistical tests or removing outliers;
- details of the software used to generate or process the data, including version number and platform.
You may be recording some of this information in a lab notebook or research journal. If so, you may find it convenient to maintain an index file that links data files to the corresponding page numbers until you have an opportunity to transfer the information into a documentation file.
A 'readme' file is a plain text file that is named 'readme' to encourage users to read it before looking at the remainder of the content. It can contain documentation directly or instruct the reader where to look to find more information. Even though it is free text, the file should be structured into sections as an aid to the reader. The following table summarises suggestions on what to include. There are some examples of readme files provided as links below the table.
| Section | What to include |
|---|---|
| Citation information |
Information needed so that the reader can cite your dataset:
|
| Methodology |
Describe how you collected the data:
|
| Third-party inputs | If you used third-party data, provide a data citation or a description of how you accessed the data. |
| Workflow |
Provide details of the steps you took to process the data:
|
| Outputs |
If your workflow generates auxiliary files as well as data files, explain which are which. Relate the outputs of your workflow to the data files you have, or will be submitting, for archiving. |
| Inventory of files |
Give the names of the files in the dataset, a short description of each, and how they interrelate. Mention related data that was not selected for inclusion, such as auxiliary files generated by your workflow. |
| File structure and conventions |
Provide details on how to interpret your data files:
|
| Licence information |
Give a short statement about the terms under which others may use the dataset. If necessary, the full text of the licence may be given in a separate plain-text file called 'licence.txt'. |
| Relationships | If applicable, give links to related datasets, alternative records or publications. |
As a researcher, the three main types of metadata you will be asked to provide are contextual metadata, discovery data, and metadata for reuse.
Contextual metadata
This describes the context within which the project was conducted. This helps to connect your data to your own research profile, and to your project, funding body and publications.
Discovery metadata
This helps other researchers to find your data, and as a result may help to increase the impact of your research. You will provide discovery metadata when you complete a record in Elements for the PEARL repository, or another research data archive or repository.
Metadata for reuse
The metadata you provide for reuse will depend on the field of your research.
- Social scientists often package their data and metadata together using DDI, or if the data are strongly statistical in nature, SDMX.
- Many biological and biomedical investigations have a corresponding Minimum Information Standard (MIBBI) setting out what information would be needed to interpret the data unambiguously and reproduce the experiment.
- Geospatial datasets are usually packaged in a format that complies with the standard ISO 19115. There are many profiles of this standard aimed at different communities; UK researchers are encouraged to use UK GEMINI, which is in turn compliant with the European INSPIRE Directive.
- Some subject-specific data archives ask for data to be submitted in a particular format. For example, the NCBI Gene Expression Omnibus (GEO) specifies a metadata set to be submitted along with data, and has developed the spreadsheet-based GEOarchive format for capturing it.
Check below for links to a number of subject-specific metadata standards and to catalogues of metadata standards.
Some subject areas have agreed on a common set of terminology to use when describing data. Metadata standards list the properties of the dataset that need to be known and vocabularies provide a standardised set of terms with which these properties can be recorded.
- The NERC Vocabulary Server provides access to many different vocabularies in use in geoscience and oceanography.
- The Open Knowledge Foundation runs the Linked Open Vocabularies service, which provides access to many different vocabularies that are suitable for use in Resource Description Framework (RDF) applications.
Resources for Writing Data Documentation (Adapted from University of Bath, 2019)
Introduction to metadata standards with a summary of commonly used standards.
Scientific American blog that provides a lay description of metadata with festive examples.
Links to resources on data documentation from the UK Data Service.
Overview of study- and data-level documentation and an overview of catalogue metadata.
Catalogue of databases relevant to science and standards used with them.
Catalogue of metadata standards by discipline (Biology, Earth Science, Physical science, Social Science & Humanities) and for general research data.
Open directory of metadata standards applicable to research data.
Data Documentation Initiative Specification. nternational metadata standard for describing social, behavioural and economics sciences data. DDI is used by the UK Data Service repository.
Minimum Information for Biological and Biomedical Investigations.This is a set of guidelines for reporting data derived by relevant methods in biosciences.
Resource for discovering metadata standards for the geosciences.
Marine Metadata Interoperability metadata standard catalogue aimed at marine science.
Metadata standards for Earth observation research.
Metadata standard for geospatial data (European directive).
Metadata standard for geospatial data.
Vocabularies used in geoscience and oceanography.
This service provides access to many different vocabularies that are suitable for use in Resource Description Framework.