Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
tagwiki:tools:formats [2017/08/05 11:22]
funkyfinwhale [Sensor data fields]
tagwiki:tools:formats [2020/01/15 12:42] (current)
funkyfinwhale
Line 1: Line 1:
 +<​page-header>​
 +====== Data formats & structures ======
 +The output of the import & export tools are individual data structures for each sensor within the tag (the Sensor data structure) and a single metadata structure (the info data structure) containing other relevant information related to the tag's deployment. (Throughout this page we use the word "​structure"​ although all the objects are technically "​lists"​ in R, and "​structures"​ in Matlab/​Octave. Also not that in R, all "​."​ in field names are replaced by "​_"​ and the items are gathered into a single list, while in Matlab there is a nested series of structure arrays.)
 +</​page-header>​
 +
 +===== Sensor data =====
 +==== Sensor data format (structure) ====
 +The basic structure for sensor data is displayed below & includes the following fields. Output is displayed as in Matlab / Octave.
 +
 +<​code>​
 +% Matlab / Octave output
 +
 +>> A = 
 +
 +  struct with fields:
 +
 +                  data: [1172115×3 double]
 +              sampling: '​regular'​
 +         ​sampling_rate:​ 32
 +    sampling_rate_unit:​ '​Hz'​
 +                 ​depid:​ '​mn12_186a'​
 +         ​creation_date:​ '​24-Jul-2017 12:​40:​14'​
 +               ​history:​ '​read_ll3m'​
 +                  name: '​A'​
 +             ​full_name:​ '​Acceleration'​
 +           ​description:​ '​triaxial acceleration'​
 +                  unit: '​1'​
 +             ​unit_name:​ '​counts'​
 +            unit_label: '​counts'​
 +           ​column_name:​ '​x,​y,​z'​
 +                 ​frame:​ '​tag'​
 +                  axes: '​FRU'​
 +                 ​files:​ '​20120704-57537-Mn-XXXX-U-NoName-Acceleration…'​
 +
 +</​code>​
 +
 +==== Sensor data fields ====
 +Sensor data field names have been chosen to be self explanatory;​ however, a more detailed description of fields can be found in the following table. ​
 +
 +|<100% 15% 15% 70%>|
 +^ Sensor name ^ Field Names ^ Description ^
 +|P, A, M, G, S, POS, T or LL | data | sensor data, either as a matrix (n by ns) or array (n by1) depending on number sensor channels|
 +|   | sampling | Specifies how the sensor data is sampled. '​regular'​ for regular sampling, i.e. constant sample interval or '​irregular'​ for data with an irregualar or inconsitent sample interval |
 +|   | sampling_rate | Sampling rate, or 1/sample interval |
 +|   | sampling_rate_unit | Sampling unit, e.g. Hz|
 +|   | depid | Deployment ID, e.g. mn12_186a. This is used to identify all records / data related to a praticular deployment |
 +|   | history |Processing chain history for current data. This will include the name of the functions from which the data were derived. For example if you have just imported the file history field will contain the name of the import tool used to create the data structure in this case read_ll3m |
 +|   | name | Abbreviated sensor name, e.g. '​A'​ |
 +|   | full_name | Full sensor name, e.g. '​Accelerometer'​ |
 +|   | description | Sensor description,​ e.g. '​triaxial acceleration'​ |
 +|   | unit_name | Engineering units, e.g. '​counts'​ |
 +|   | unit_label | Unit label |
 +|   | column_name | Data column names, e.g. '​x,​y,​z'​ |
 +|   | frame | Data frame, '​tag'​ or '​animal'​ frame. |
 +|   | axes | Axis map, e.g. '​FRU'​ i.e. '​Front,​ Right & Up' |
 +|   | files | List of files used to generate the sensor data structure |
 +
 +
 +====Sensor data output====
 +<​code>​
 +% Matlab / Octave output. Command displays the first 10 rows of accelerometer data (x,y,z axes)
 +
 +>> A.data(1:​10,:​) ​          
 +
 +ans =
 +
 +    8.5263 ​   4.4584 ​   2.0690
 +    8.4906 ​   4.3222 ​   2.3321
 +    8.5974 ​   4.2704 ​   1.8021
 +    8.7186 ​   4.2414 ​   0.8304
 +    8.7263 ​   4.2204 ​   0.2391
 +    8.5999 ​   4.3505 ​   0.5508
 +    8.4265 ​   4.7551 ​   1.4347
 +    8.3038 ​   5.3248 ​   2.0671
 +    8.2610 ​   5.7892 ​   2.0696
 +    8.2566 ​   6.0020 ​   1.8019
 +</​code>​
 +
 +==== Sensor names, axes & units ====
 +The Sensor definition table (sensor_names.csv) lives in the I/O toolbox, this contains definitions for the following Sensors by default, but it can be edited ​ & updated by the user to include novel sensors.
 +^Sensor name (name)^Abbreviated name (abbrev)^ Description (description) ^Number of axes (axes)^ Units (def_units)^Units name (def_unit_name) ^Units Label (def_label)^Frame (def_frame)^Axes (def_axes)^Data columns (def_cols)^
 +|Pressure|P|dive depth|1|m H2O|meters H2O (salt)|meters||D|
 +|Depth|P|dive depth|1|m H2O|meters H2O (salt)|meters||D|
 +|Altitude|P|altitude above sea-level|1|m|meters above sea-level|meters||U|
 +|Acceleration|A|triaxial acceleration|3|m/​s2|meters per seconds squared|m/​s<​sup>​2</​sup>​|tag|FRU|"​x,​y,​z"​|
 +|Magnetometer|M|triaxial magnetometer|3|uT|micro Tesla|μT|tag|FRU|"​x,​y,​z"​|
 +|Gyroscope|G|triaxial gyroscope|3|rads/​s|radians per second|rads/​s|tag|FRU|"​x,​y,​z"​|
 +|Speed|S|speed with respect to medium|1|m/​s|meters per second|m/​s||F|
 +|Position|POS|GPS position|2|degrees|decimal degrees|degrees|WGS84|NE|"​lat,​long"​|
 +|Temperature|T|temperature|1|degrees C|degrees Celsius|°C|||
 +|Light|LL|light level|1|lux|lux|lux|||
 +
 +===== Metadata ​ =====
 +Metadata is stored in a comma separated file (csv) that is machine readable by text editors, Office packages as a spreadsheet and Matlab, Octave and R. In Matlab, Octave & R  it may be stored as a data structure <wrap em>​info</​wrap>​.  ​
 +==== info data structure ====
 +<WRAP center round alert 75%>
 +Page under construction. Information may change!
 +</​WRAP>​
 +
 +====Info data output====
 +
 +<​code>​
 +% Matlab / Octave output
 +
 +info = 
 +
 +  struct with fields:
 +
 +                             ​depid:​ '​bp12_196a'​
 +                       ​data_source:​ '​20120714-57536-Bp-Fxxx-U-Nameless-Acceleration-X.TXT,​20120714-57536-Bp-Fxxx-U-Nameless-Acceleration-Y.TXT,​20120714-57536-Bp-Fxxx-U-Nameless-Acceleration-Z.TXT,​20120714-57536-Bp-Fxxx-U-Nameless-Compass-X.TXT,​20120714-57536-Bp-Fxxx-U-Nameless-Compass-Y.TXT,​20120714-57536-Bp-Fxxx-U-Nameless-Compass-Z.TXT,​20120714-57536-Bp-Fxxx-U-Nameless-Depth.TXT,​20120714-57536-Bp-Fxxx-U-Nameless-Propeller.TXT,​20120714-57536-Bp-Fxxx-U-Nameless-Temperature.TXT'​
 +                       ​data_nfiles:​ 9
 +                       ​data_format:​ '​csv'​
 +                     ​device_serial:​ '​57536'​
 +                       ​device_make:​ '​Little Leonardo, Japan'
 +                       ​device_type:​ '​Archival'​
 +                 ​device_model_name:​ '​UWE-3MPD3GT'​
 +              device_model_version:​ '​W2000'​
 +                        device_url: '​http://​l-leo.com/​eng/​data-logger'​
 +                      sensors_list:​ '3 axis Accelerometer,​3 axis Magnetometer,​Pressure,​Temperature,​Speed'​
 +              dephist_device_tzone:​ 'UTC -4'
 +             ​dephist_device_regset:​ '​dd-mm-yyyy HH:​MM:​SS'​
 +     ​dephist_device_datetime_start:​ '​14-Jul-2012 12:​35:​01'​
 +                  sensors_firmware:​ 'Not specified'​
 +                  sensors_software:​ 'Not specified'​
 +             ​animal_species_common:​ 'Fin whale'
 +            animal_species_science:​ '​Balaenoptera physalus'​
 +               ​animal_database_url:​ '​https://​www.rorqual.com/'​
 +           ​dephist_deploy_locality:​ '​Mingan Islands, Anticosti & surrounding sea areas, Gulf of St. Lawrence, Canada. (UTC -4).'
 +       ​dephist_deploy_location_lat:​ NaN
 +       ​dephist_deploy_location_lon:​ NaN
 +     ​dephist_deploy_datetime_start:​ '​14-Jul-2012 13:​39:​10'​
 +             ​dephist_deploy_method:​ '​Suction cups'
 +                      project_name:​ '​SENSES2011'​
 +            project_datetime_start:​ '​01-Jul-2011'​
 +              project_datetime_end:​ '​01-Jan-2016'​
 +                     ​provider_name:​ 'René Swift'
 +                  provider_details:​ 'Sound & Movement Tags Group, SMRU, University of St. Andrews'​
 +                    provider_email:​ '​rjs30@st-andrews.ac.uk'​
 +                  provider_license:​ '​Contact data provider'​
 +                     ​provider_cite:​ '​Contact data provider'​
 +                      provider_doi:​ '​Contact data provider'​
 +                        udm_export: 1
 +                udm_depid_original:​ '​20120714-57536'​
 +                udm_datetime_units:​ '​UTC'​
 +      udm_dephist_release_datetime:​ '​2012-07-14 19:​00:​59'​
 +           ​udm_dephist_release_lat:​ NaN
 +           ​udm_dephist_release_lon:​ NaN
 +    udm_dephist_retrieval_datetime:​ '​2012-07-14 19:​03:​49'​
 +         ​udm_dephist_retrieval_lat:​ NaN
 +         ​udm_dephist_retrieval_lon:​ NaN
 +                   ​udm_animal_name:​ '​Nameless'​
 +                    udm_animal_sex:​ '​Unknown'​
 +                      magfield_lon:​ -64.1000
 +                      magfield_lat:​ 50.1000
 +                     ​creation_date:​ '​05-Jul-2018 12:​04:​43'​
 +</​code>​
 +
 +====Info data fields====
 +
 +|<100% 15% 42.5% 42.5%>|
 +^ Field                    ^ Help / Description ​                                                   ^ Example ​             ^
 +| info.depid ​              | Deployment ID  eg.mn12_126a, ​ Used to link tag metadata to tag data   | mn12_186a ​           |
 +| info.data.source ​        | Source data file or files name, e.g. mn14_35a.dtg ​ or mn12_126a.obj. ​ | mn12_186a.obj ​       |
 +| info.data.datetime.made ​ | Date time of creation/​modification of the meta data document. ​        | 2017-06-24 09:​29:​30 ​ |
 +| info.data.nfiles ​        | Number of source files. ​                                              | 1                    |
 +
 +=== Device information ===
 +|<100% 15% 42.5% 42.5%>|
 +^ Field                    ^ Help / Description ​                                                   ^ Example ​             ^
 +|info.data.format|Format of source data files eg. CSV, BIN, WAV, DTG, OBJ,  TXT. Data derived. |OBJ|
 +|info.device.serial|Device serial number.|57537|
 +|info.device.make|Device manufacturer.|Little Leonardo|
 +|info.device.type|Type of device can only take the following values: archival pop-up satellite acoustic tag or acoustic receiver.|Archival|
 +|info.device.model|Model name and version e.g. 3MPD3GT-W2000 or  D4W|3MPD3GT-W2000|
 +|info.device.url|URL to tag model specifications.|http://​l-leo.com/​eng/​data-logger|
 +
 +=== Sensor information ===
 +|<100% 15% 42.5% 42.5%>|
 +^ Field                    ^ Help / Description ​                                                   ^ Example ​             ^
 +| info.sensors.firm ​ | Device firmware version. ​                                                                                                                                      | Not specified ​                                                                     |
 +| info.sensors.soft ​ | Software off loader version in case there are known issues with a particular version that should be taken into account ​                                        | Not specified ​                                                                     |
 +| info.sensors.list ​ | List of sensors on each tag e.g. Pressure, Temperature,​ Light, Fluorescence,​ Position (Argos, GPS Geo-location) Sonar – pull down menu tick box. Data derived ​ |‘3axis Accelerometer’,​ ‘3 axis Magnetometer’,​’Pressure’,​’Speed’,​ ‘Temperarature’ ​ |
 +
 +=== Deployment information===
 +
 +== Tag & Animal information ==
 +|<100% 15% 42.5% 42.5%>|
 +^ Field                    ^ Help / Description ​                                                   ^ Example  ​
 +|info.animal.id|Catalogue ID, Band ID or Flipper Tag ID. Optional|Unknown|
 +|info.animal.species.common|Common or vernacular name for species|Humpback whale|
 +|info.animal.species.science|Scientific name|Megaptera novaeangliae|
 +|info.animal.dbase.url|URL to organisation holding photo-id, tag & release data,  linked to animal id. Required if info.animal.id is filled in|https://​www.rorqual.com/​|
 +
 +==Timezone & Time information==
 +|<100% 15% 42.5% 42.5%>|
 +^ Field                    ^ Help / Description ​                                                   ^ Example ​             ^
 +|info.dephist.device.tzone|Time zone of device (+- HRS UTC). |-4|
 +|info.dephist.device.regset|Regional settings for device specifically date format|dd/​mm/​yyyy HH:MM:SS|
 +|info.dephist.device.datetime.start| Device start date and time, this may be different to the deployment datetime|2012-07-04 07:59:01|
 +
 +==Locality information==
 +|<100% 15% 42.5% 42.5%>|
 +^ Field                    ^ Help / Description ​                                                   ^ Example ​             ^
 +|info.dephist.deploy.locality|Locality State/​Territory Country of Deployment|Mingan Islands, Gulf of St. Lawrence, Quebec, Canada|
 +|info.dephist.deploy.location.lat|Latitude of Deployment ​ (-90s to +90N decimal degrees) |50.1748123 |
 +|info.dephist.deploy.location.lon|Longitude of Deployment ​ (-180W to +180E decimal degrees) |-64.0737686|
 +|info.dephist.deploy.datetime.start|UTC Date & Time of Deployment|2012-07-04 12:42:21|
 +|info.dephist.deploy.method|Describe how the tag was attached (e.g. glued, implanted, suction cups).|Suction cup|
 +
 +===Project information===
 +|<100% 15% 42.5% 42.5%>|
 +^ Field                    ^ Help / Description ​                                                   ^ Example ​             ^
 +|info.project.name|Project Name|SENSES-2012|
 +|info.project.datetime|Start & End time for all project related deployments|2012-06-01 : 2012-10-01|
 +
 +
 +===Provider information===
 +|<100% 15% 42.5% 42.5%>|
 +^ Field                    ^ Help / Description ​                                                   ^ Example ​             ^
 +|info.provider.name|Name of data provider / license holder|René Swift|
 +|info.provider.details|Affiliation etc|Sea Mammal Research Unit|
 +|info.provider.email|Email of data provider|rjs30@st-andrews.ac.uk|
 +|info.provider.license|Describe the data provider restrictions to data access and distribution. ODC-ODbl, CC-by|Contact data provider|
 +
 +===Citation information===
 +|<100% 15% 42.5% 42.5%>|
 +^ Field                    ^ Help / Description ​                                                   ^ Example ​             ^
 +|info.provider.cite|Citation to be used in publications |Contact data provider|
 +|info.provider.doi|DOI for data|Contact data provider|
 +|info.udm.export|Binary switch specifying if the User  wants their user defined meta-data to published along with the rest of standard metadata. ​ |1|
 +
 +
 +