Data formats & structures

Book Creator

Add this page to your book

Book Creator

Remove this page from your book

Manage book (0 page(s))

Help

The output of the import & export tools are individual data structures for each sensor within the tag (the Sensor data structure) and a single metadata structure (the info data structure) containing other relevant information related to the tag's deployment. (Throughout this page we use the word “structure” although all the objects are technically “lists” in R, and “structures” in Matlab/Octave. Also not that in R, all “.” in field names are replaced by “_” and the items are gathered into a single list, while in Matlab there is a nested series of structure arrays.)

Sensor data format (structure)

The basic structure for sensor data is displayed below & includes the following fields. Output is displayed as in Matlab / Octave.

% Matlab / Octave output

>> A = 

  struct with fields:

                  data: [1172115×3 double]
              sampling: 'regular'
         sampling_rate: 32
    sampling_rate_unit: 'Hz'
                 depid: 'mn12_186a'
         creation_date: '24-Jul-2017 12:40:14'
               history: 'read_ll3m'
                  name: 'A'
             full_name: 'Acceleration'
           description: 'triaxial acceleration'
                  unit: '1'
             unit_name: 'counts'
            unit_label: 'counts'
           column_name: 'x,y,z'
                 frame: 'tag'
                  axes: 'FRU'
                 files: '20120704-57537-Mn-XXXX-U-NoName-Acceleration…'

Sensor data fields

Sensor data field names have been chosen to be self explanatory; however, a more detailed description of fields can be found in the following table.

Sensor name	Field Names	Description
P, A, M, G, S, POS, T or LL	data	sensor data, either as a matrix (n by ns) or array (n by1) depending on number sensor channels
	sampling	Specifies how the sensor data is sampled. 'regular' for regular sampling, i.e. constant sample interval or 'irregular' for data with an irregualar or inconsitent sample interval
	sampling_rate	Sampling rate, or 1/sample interval
	sampling_rate_unit	Sampling unit, e.g. Hz
	depid	Deployment ID, e.g. mn12_186a. This is used to identify all records / data related to a praticular deployment
	history	Processing chain history for current data. This will include the name of the functions from which the data were derived. For example if you have just imported the file history field will contain the name of the import tool used to create the data structure in this case read_ll3m
	name	Abbreviated sensor name, e.g. 'A'
	full_name	Full sensor name, e.g. 'Accelerometer'
	description	Sensor description, e.g. 'triaxial acceleration'
	unit_name	Engineering units, e.g. 'counts'
	unit_label	Unit label
	column_name	Data column names, e.g. 'x,y,z'
	frame	Data frame, 'tag' or 'animal' frame.
	axes	Axis map, e.g. 'FRU' i.e. 'Front, Right & Up'
	files	List of files used to generate the sensor data structure

Sensor data output

% Matlab / Octave output. Command displays the first 10 rows of accelerometer data (x,y,z axes)

>> A.data(1:10,:)           

ans =

    8.5263    4.4584    2.0690
    8.4906    4.3222    2.3321
    8.5974    4.2704    1.8021
    8.7186    4.2414    0.8304
    8.7263    4.2204    0.2391
    8.5999    4.3505    0.5508
    8.4265    4.7551    1.4347
    8.3038    5.3248    2.0671
    8.2610    5.7892    2.0696
    8.2566    6.0020    1.8019

Sensor names, axes & units

The Sensor definition table (sensor_names.csv) lives in the I/O toolbox, this contains definitions for the following Sensors by default, but it can be edited & updated by the user to include novel sensors.

Sensor name (name)	Abbreviated name (abbrev)	Description (description)	Number of axes (axes)	Units (def_units)	Units name (def_unit_name)	Units Label (def_label)	Frame (def_frame)	Axes (def_axes)	Data columns (def_cols)
Pressure	P	dive depth	1	m H2O	meters H2O (salt)	meters		D
Depth	P	dive depth	1	m H2O	meters H2O (salt)	meters		D
Altitude	P	altitude above sea-level	1	m	meters above sea-level	meters		U
Acceleration	A	triaxial acceleration	3	m/s2	meters per seconds squared	m/s²	tag	FRU	“x,y,z”
Magnetometer	M	triaxial magnetometer	3	uT	micro Tesla	μT	tag	FRU	“x,y,z”
Gyroscope	G	triaxial gyroscope	3	rads/s	radians per second	rads/s	tag	FRU	“x,y,z”
Speed	S	speed with respect to medium	1	m/s	meters per second	m/s		F
Position	POS	GPS position	2	degrees	decimal degrees	degrees	WGS84	NE	“lat,long”
Temperature	T	temperature	1	degrees C	degrees Celsius	°C
Light	LL	light level	1	lux	lux	lux

Metadata is stored in a comma separated file (csv) that is machine readable by text editors, Office packages as a spreadsheet and Matlab, Octave and R. In Matlab, Octave & R it may be stored as a data structure info.

info data structure

Page under construction. Information may change!

Info data output

% Matlab / Octave output

info = 

  struct with fields:

                             depid: 'bp12_196a'
                       data_source: '20120714-57536-Bp-Fxxx-U-Nameless-Acceleration-X.TXT,20120714-57536-Bp-Fxxx-U-Nameless-Acceleration-Y.TXT,20120714-57536-Bp-Fxxx-U-Nameless-Acceleration-Z.TXT,20120714-57536-Bp-Fxxx-U-Nameless-Compass-X.TXT,20120714-57536-Bp-Fxxx-U-Nameless-Compass-Y.TXT,20120714-57536-Bp-Fxxx-U-Nameless-Compass-Z.TXT,20120714-57536-Bp-Fxxx-U-Nameless-Depth.TXT,20120714-57536-Bp-Fxxx-U-Nameless-Propeller.TXT,20120714-57536-Bp-Fxxx-U-Nameless-Temperature.TXT'
                       data_nfiles: 9
                       data_format: 'csv'
                     device_serial: '57536'
                       device_make: 'Little Leonardo, Japan'
                       device_type: 'Archival'
                 device_model_name: 'UWE-3MPD3GT'
              device_model_version: 'W2000'
                        device_url: 'http://l-leo.com/eng/data-logger'
                      sensors_list: '3 axis Accelerometer,3 axis Magnetometer,Pressure,Temperature,Speed'
              dephist_device_tzone: 'UTC -4'
             dephist_device_regset: 'dd-mm-yyyy HH:MM:SS'
     dephist_device_datetime_start: '14-Jul-2012 12:35:01'
                  sensors_firmware: 'Not specified'
                  sensors_software: 'Not specified'
             animal_species_common: 'Fin whale'
            animal_species_science: 'Balaenoptera physalus'
               animal_database_url: 'https://www.rorqual.com/'
           dephist_deploy_locality: 'Mingan Islands, Anticosti & surrounding sea areas, Gulf of St. Lawrence, Canada. (UTC -4).'
       dephist_deploy_location_lat: NaN
       dephist_deploy_location_lon: NaN
     dephist_deploy_datetime_start: '14-Jul-2012 13:39:10'
             dephist_deploy_method: 'Suction cups'
                      project_name: 'SENSES2011'
            project_datetime_start: '01-Jul-2011'
              project_datetime_end: '01-Jan-2016'
                     provider_name: 'René Swift'
                  provider_details: 'Sound & Movement Tags Group, SMRU, University of St. Andrews'
                    provider_email: 'rjs30@st-andrews.ac.uk'
                  provider_license: 'Contact data provider'
                     provider_cite: 'Contact data provider'
                      provider_doi: 'Contact data provider'
                        udm_export: 1
                udm_depid_original: '20120714-57536'
                udm_datetime_units: 'UTC'
      udm_dephist_release_datetime: '2012-07-14 19:00:59'
           udm_dephist_release_lat: NaN
           udm_dephist_release_lon: NaN
    udm_dephist_retrieval_datetime: '2012-07-14 19:03:49'
         udm_dephist_retrieval_lat: NaN
         udm_dephist_retrieval_lon: NaN
                   udm_animal_name: 'Nameless'
                    udm_animal_sex: 'Unknown'
                      magfield_lon: -64.1000
                      magfield_lat: 50.1000
                     creation_date: '05-Jul-2018 12:04:43'

Info data fields

Field	Help / Description	Example
info.depid	Deployment ID eg.mn12_126a, Used to link tag metadata to tag data	mn12_186a
info.data.source	Source data file or files name, e.g. mn14_35a.dtg or mn12_126a.obj.	mn12_186a.obj
info.data.datetime.made	Date time of creation/modification of the meta data document.	2017-06-24 09:29:30
info.data.nfiles	Number of source files.	1

Device information

Field	Help / Description	Example
info.data.format	Format of source data files eg. CSV, BIN, WAV, DTG, OBJ, TXT. Data derived.	OBJ
info.device.serial	Device serial number.	57537
info.device.make	Device manufacturer.	Little Leonardo
info.device.type	Type of device can only take the following values: archival pop-up satellite acoustic tag or acoustic receiver.	Archival
info.device.model	Model name and version e.g. 3MPD3GT-W2000 or D4W	3MPD3GT-W2000
info.device.url	URL to tag model specifications.	http://l-leo.com/eng/data-logger

Sensor information

Field	Help / Description	Example
info.sensors.firm	Device firmware version.	Not specified
info.sensors.soft	Software off loader version in case there are known issues with a particular version that should be taken into account	Not specified
info.sensors.list	List of sensors on each tag e.g. Pressure, Temperature, Light, Fluorescence, Position (Argos, GPS Geo-location) Sonar – pull down menu tick box. Data derived	‘3axis Accelerometer’, ‘3 axis Magnetometer’,’Pressure’,’Speed’, ‘Temperarature’

Deployment information

Tag & Animal information

Field	Help / Description
info.animal.id	Catalogue ID, Band ID or Flipper Tag ID. Optional	Unknown
info.animal.species.common	Common or vernacular name for species	Humpback whale
info.animal.species.science	Scientific name	Megaptera novaeangliae
info.animal.dbase.url	URL to organisation holding photo-id, tag & release data, linked to animal id. Required if info.animal.id is filled in	https://www.rorqual.com/

Timezone & Time information

Field	Help / Description	Example
info.dephist.device.tzone	Time zone of device (+- HRS UTC).	-4
info.dephist.device.regset	Regional settings for device specifically date format	dd/mm/yyyy HH:MM:SS
info.dephist.device.datetime.start	Device start date and time, this may be different to the deployment datetime	2012-07-04 07:59:01

Locality information

Field	Help / Description	Example
info.dephist.deploy.locality	Locality State/Territory Country of Deployment	Mingan Islands, Gulf of St. Lawrence, Quebec, Canada
info.dephist.deploy.location.lat	Latitude of Deployment (-90s to +90N decimal degrees)	50.1748123
info.dephist.deploy.location.lon	Longitude of Deployment (-180W to +180E decimal degrees)	-64.0737686
info.dephist.deploy.datetime.start	UTC Date & Time of Deployment	2012-07-04 12:42:21
info.dephist.deploy.method	Describe how the tag was attached (e.g. glued, implanted, suction cups).	Suction cup

Project information

Field	Help / Description	Example
info.project.name	Project Name	SENSES-2012
info.project.datetime	Start & End time for all project related deployments	2012-06-01 : 2012-10-01

Provider information

Field	Help / Description	Example
info.provider.name	Name of data provider / license holder	René Swift
info.provider.details	Affiliation etc	Sea Mammal Research Unit
info.provider.email	Email of data provider	rjs30@st-andrews.ac.uk
info.provider.license	Describe the data provider restrictions to data access and distribution. ODC-ODbl, CC-by	Contact data provider

Citation information

Field	Help / Description	Example
info.provider.cite	Citation to be used in publications	Contact data provider
info.provider.doi	DOI for data	Contact data provider
info.udm.export	Binary switch specifying if the User wants their user defined meta-data to published along with the rest of standard metadata.	1