need a database for wich fix_pressure does not fix all the pressure offsets, and so fix_offset_pressure needs to be use.
Usage
In this vignette we will learn how to check depth data and correct it using the functions fix_pressure() and fix_offset_pressure(). You will also learn how to remove off-animal tag data. For the full theoretical description, refer to depth checks and corrections – technical explanation.
[pf,pc] = fix_pressure(P,T,sampling_rate,maxp)
[pf,pc] <- fix_pressure(P,T,sampling_rate,maxp)
Inputs
p:is s sensor list or vector of depth/altitude in meters.t: is a sensor list or vector of temperature in degrees Celsius.sampling_rate:the sampling rate of the sensor data in Hz (samples per second). This is only needed if p and t are not a sensor lists. The depth and temperature must both have the same sampling rate (usedecdc()if needed to achieve this).-
maxp:is the maximum depth or altitude reading in the pressure data for which the animal could actually be at the surface. This is a rough measurement of the potential error in the pressure data. The unit is meters. Start with a small value, e.g., 2m and rerunfix_pressurewith a larger value if there are still obvious temperature-related errors in the resulting depth/altitude profile.
Outputs
pf:is a sensor list, structure or vector of corrected depth/altitude measurements at the same sampling rate as the input data. If the input is a sensor list, the output will also be.pc:is a list or structure containing the pressure offset and temperature correction coefficients. It has fields:pc$tref(pc.trefin Matlab) which is the temperature compensation polynomial. This is used within the function to correct pressure as follows:p + stats::polyval(pc$tcomp, t - pc$tref), (p = p+polyval(pc.tcomp,t-pc.tref)in Matlab).
For a full description on the usage of fix_offset_pressure(), refer to depth checks and corrections – technical explanation.
Theoretical Description
Biologging data are not always ready-to-use when you read them off the tag. Some corrections may be needed. Quality checking is fairly easy for pressure data from aquatic mammals because we know they will breathe at the surface. Pressure sensors tend to drift with temperature, so the pressure calibration needs the temperature data in order to correct this drift. For a more detailed theoretical description, refer to depth checks and corrections – technical explanation.
Technical Description
In this vignette, you will learn how to correct a depth or altutide profile for offsets caused by miscalibration and temperature using fix_pressure(). If there are still pressure offsets after correcting for temperature using fix_pressure(), you should use the funcntion fix_offset_pressure(). For a more detailed technical description, refer to depth checks and corrections – technical explanation.
Verification
You could simply verify that both fix_pressure()and fix_offset_pressure() works by simply plotting the the original depth, p and the fixed depth, pf, and check that the top of the dive profile, when the animal is at the surface, is flatter and close to zero. (How would this work for a flying animal? how could you check that is working? for a seabird that rests in the water would be the same but what for a flying animal that rests at different altitudes) Stacy says: maybe just note this — it will only work if the tagged animal returns to a consistent known depth or altitude to rest or breathe? For a full description on how to run the verification, refer to depth checks and corrections – technical explanation.
Caveats
Both functions fix_pressure()and fix_offset_pressure() make a number of assumptions about the depth/altitude data and about the behaviour of animals. For example, the depth data should have few incorrect outlier (negative) values that fall well beyond the surface. For a full description of caveats, refer to depth checks and corrections – technical explanation.
Load sensor structure calibrated data and the calibration file
In the calibrated data structure tutorial, we learned how calibrate the pressure, temperature, accelerometer and magnetometer data and saved an ncfile for the deployment.
Load the test dataset XXX, which belongs to the data recorded from a xxx. This dataset is built into the tagtools package, so you can access it using system.file.
tag='xxx';
ncname = [tag 'sens25cal']
xx_file_path = xxx
xx = load_nc(xx_file_path)
MN_file_path <- system.file("extdata", "mn12_186a_raw.nc", package = "tagtools", mustWork = TRUE)
MN <- load_nc(MN_file_path)
Then use plott() to inspect the depth/altitude, and temperature:
figure
plott(xx.P, xx.T)
subplot(211)
ylabel('Depth')
subplot(212)
ylabel('Temperature')
plott(X = list(Depth = MN$P, Temperature= MN$T), r = c(TRUE,FALSE))
# r=c(TRUE,FALSE) tells plot to reverse the y-axis for the Depth data (so that it looks like a dive profile), but not for the Temperature data (which would be silly). )
Expand to show figure …
Look in info to find what species the data come from. Are the depth values reasonable for this species?
The data belongs to a humpback whale. And so the depth values up to 80 meters are reasonable for this species. However, there are outliers in the depth data.
As with all raw depth data, there are some problems with this dive profile. See if you can find evidence for each of these in the depth and temperature plot:
1) Incorrect calibration of the sensor
1. Pay attention to the depths reported… While it looks like a dive profile, mostly, what looks like the “surface” is actually around some −10m (that is, ten meters high in the air)! We know that humpback whales do not, in fact, go ten meters in the sky routinely. This suggests the sensor has not been well-calibrated.
2) Occasional outliers
Occasionally, there are some peaks that go higher yet than these ten meters high in the sky momentarily (more like 15 meters). These are outliers.
3) Coarse depth resolution
Zoom in to the shallow part of the dive to notice this. The individual measurements are only precise to the meter; thus it will appear to jump back and forth more suddenly than we would expect for a relaxed whale at the surface.
4) Temperature sensitivity
The temperature gradually rises as the recording nears the end, then spikes down and up. This is accompanied by a gradually decreasing depth from ten meters in the sky to the surface. Together, the two trends strongly suggest that the depth data has been affected by the changing temperature as the tag sits near the surface.
What to do about periods of data when the tag is not on the animal
Not all tags have a way to start logging as soon as the tag has been deployed on the animal. Often data logging is started by a time trigger or alarm, and the researcher has to make a guess as to when the tag will be deployed to set its start time appropriately. Often this means that a tag is logging data before it is put on an animal.
Equally, tags have no means of detecting when they release from the tagged animal. As a consequence, they may continue to log data after they release. In most cases, the logged data from before and after deployment has no use. To reduce the data to just the periods when the tag is on the animal, use the tool crop:
Expand to show figure …
#> [1] "Position your cursor and then click once followed by clicking FINISH to change the start, or click twice in the
# same spot followed by clicking FINISH to change the end. If you wish to change both the start and end click once at the
# start time desired and twice at the end time desired."
This displays an interactive depth plot (It does not know to reverse the y-axis, so that’s why it looks upside down.). Follow the instructions to select the obvious diving section of the data and then click finish. The function returns a new data structure which contains just the selected part of the dive profile.
Once you’ve cropped the depth data to the relevant segment, this will be stored as Pc. Now, use plott to plot Pc to make sure you cropped it correctly.
Expand to show figure …
If all has gone well, your plott should be absolutely beautiful.
As an alternative to crop, if you already know the start and end times (with appropriate units) that you would like to crop the data to, you can use crop_to. For instance, to get back something like what you just did with crop from the same depth data:
The resulting sensor data list also contains fields that document what you just did. To see them:
Expand to show terminal output …
List of 21 $ data : num [1:721] -7.5 -7 -7.5 -7.5 -7.5 -7 -7.5 -7 -7 -7 ... $ sampling : chr "regular" $ sampling_rate : num 1 $ sampling_rate_unit: chr "Hz" $ depid : chr "mn12_186a" $ creation_date : chr "07-Aug-2017 14:51:49" $ history : chr "read_ll3m,crop_to" $ type : chr "press" $ full_name : chr "Pressure" $ description : chr "dive depth" $ unit : chr "m H2O" $ unit_name : chr "meters H2O (salt)" $ unit_label : chr "meters" $ start_offset : num 0 $ start_offset_units: chr "second" $ axes : chr "D" $ files : chr "20120704-57537-Mn-XXXX-U-NoName-Depth.TXT" $ crop : num [1:2] 1500 2220 $ crop_units : chr "seconds" $ start_time : num 1500 $ start_time_units : chr "seconds"
The history field keeps track of the operations that you perform on a data structure. This helps with traceability if you make the processed data available in an archive. The crop and start_time fields show how the original data was changed: the start_time is with respect to the field ‘dephist_device_datetime_start’ in the info structure which says when the tag recording started. Use plott to plot Pc to make sure you cropped it correctly:
[Note]The results from here on out will look a little different depending on where exactly you’ve cropped your data, but it should come to essentially the same thing..
Removing outliers
Outliers or spikes in the data may result from errors in the tag or poor sensor performance under rapidly changing environmental conditions. For example in this data set, rapid changes in temperature and pressure as the animal surfaces cause small glitches in the data. These are not representative of the animal’s behaviour so we need to remove them. A good way to do this is with a median filter. To find out how this function works, type:
This can be called using:
Your variable Pcm now contains the median-filtered, cropped depth data. Check its history to verify that the median filtering has been added. Compare it against the unfiltered data using [Note]This plots Pc in the upper panel and Pcm in the lower one, and reverses the y-axis labeling on both plots (by setting r to TRUE both times).:
plott(Pc, Pcm)
subplot(211)
ylabel('Depth')
subplot(212)
ylabel('Depth median filter')
plott(X=list(Pc=Pc, Pcm=Pcm), r=c(TRUE,TRUE))
Expand to show figure …
Correcting pressure offsets & temperature effects
The next step is to correct the ‘0’ pressure offset of the depth sensor (so that the animal is not 10 m out of the water when it is really at the surface). We can also compensate for temperature at the same time. To do this we have to first crop the temperature data to match the pressure data. You can do this using:
This uses the crop information stored in Pc to do the same operation on T. Plot them together to confirm that the time intervals (horizontal ticks) are the same:
plott(Pc, Tc)
subplot(211)
ylabel('Depth')
subplot(212)
ylabel('Temperature')
plott(X=list(Pcm=Pcm, Tc=Tc), r=c(TRUE,FALSE))
Expand to show figure …
The tag toolbox has a function to correct pressure data called ‘fix_pressure’. To find out what it does and what assumptions it makes about the data, type:
Use this function by calling:
Compare the compensated dive profile to the uncompensated cropped ones using plott. [Note]The code below will produce three plots, which might make them a bit small. Hit “Zoom” to view them in a larger window.
plott(Pc, Pcm, Pcmf)
subplot(311)
ylabel('Depth')
subplot(312)
ylabel('Depth median filter')
subplot(313)
ylabel('Depth fixed')
plott(X=list(Pc=Pc, Pcm=Pcm, Pcmf=Pcmf$p), r=c(TRUE,TRUE,TRUE))
Expand to show figure …
Which of the problems that we listed above have been taken care of? Any ideas what you could do about the remaining one(s)?
Incorrect calibration of the sensor and temperature sensitivity have both been addressed by now, since the whale is actually resting and breathing at the surface, rather than ten meters in the air. Additionally, some of the occasional outliers have been removed by the median filter.
However, none of this has increased the resolution of the depth data. In order to resolve this last issue, ultimately, new data would have to be taken at higher resolution.
we need an example in which Pcmf is still no good so we show how to use fix_offset_pressure().
In our current example we can see that Pf still doesn’t look correct when the animal surfaces and the ‘0’ pressure seems to be changing over time. Thus, we need to run the other tool, fix_offset_pressure().
[pf,poffs] = fix_offset_pressure(xx.P,intvl,tc)
Pf <- fix_offset_pressure(xx$P,intvl,tc);
The two numbers intvl, and tc control how fix_offset_pressure() searches for potential surfacing intervals and these need to be adjusted to match the behavior of your animal. The first number, intvl, is the search interval in seconds that is used to find surfacings. This should be chosen to be a little more than the usual dive duration. The second number, tc , controls how quickly the zero-depth offset is allowed to change. This should normally be several times larger than the first number (i.e., preventing the offset from changing quickly) unless the plot of Pc suggests that the depth sensor is making steps in its zero offset (this could happen if the sensor gets hit or gets some sand wedged on to the diaphragm). Plot Pf and adjust the two numbers in fix_offset_pressure() up or down as required to make the surfacings look reasonable.
First, get the statistics for the dives so as to set intvl and tc values. Use the function find_dives() (check find-dives tutorial for a more detail on how this function works) for this:
ADD CODE HERE
mindepth=5
d = finddives(xx.P.data,fs,mindepth)
divedur=d(:,2)-d(:,1)
mean(divedur)%mean dive duration in seconds.
The value for the intvl should be higher than the mean dive duration in seconds. Then set the tc value to at least twice the value of the intvl . Then plot Pf and check if the offset has been improved. If not try other values for tc until the depth at the surface is close to zero:
interv=
tc=
Pf = fix_offset_pressure(xx.P,intvl,tc);
plott(xx.P, Pf)
subplot(211)
ylabel('Depth')
subplot(212)
ylabel('Fixed depth')
interv<-
tc<-
Pf <- fix_offset_pressure(xx$P,intvl,tc);
plott(X = list(`Depth` = xx$P$data, `Fixed depth` = Pf$data), fsx = sampling_rate)
Expand to show figure …
Saving the results
When you are comfortable that the data pass the quality checks, add them to the NetCDF file that you have loaded at the beginning of this tutorial with all the calibrated sensor data, using add_nc() [Note]If the initial calibration was fine and you didn’t need to run the correction tools, then you do not need to save anything..
The corrections that you made to get Pf are stored in its structure so this information will also be saved automatically in your nc file – verify this by showing the contents of Pf, by simply typing Pf on the command window in Matlab (the alternative is shown for R).
Your processing steps are listed in the ‘history’ field and so are traceable as you would expect in a professional archive.
Common Questions
No questions come to my mind… any suggestion? Stacy says: Maybe, “Is this needed (and will it work) for all tag types? A. Some tag types from Wildlife Computers and CATS (and maybe others) do some version of this depth-offset or temperature-depth correction automatically, before the data are available to the user. In that case, applying these tools may provide further improvement, or may make any remaining issues worse – be sure to verify that the results make sense before using output from these tools, especially if your tag already does onboard depth data correction of some kind.”
Further Resources
For a more in-depth explanation of the usage and inner workings of the fix_pressure() and fix_offset_pressure() (need to add fix_offset_pressure in the repository Stacy says: again, for some reason there is not an R version of this, I guess I need to make one and document it so that the help file will show up on the repo) tools, refer to the Technical Explanation of depth checks and corrections.
If you’d like to continue working through these vignettes, tag-to-whale-frame is probably a good option. It deals with estimating and correcting the orientation of a tag on an animal. This vignette will use tag data that are not aligned to the animal’s body axes, e.g., because the tag was applied to a free-moving whale.
Also, dive-stats (ADD LINK) and find-dives might be a good option if you’re especially interested in analyzing individual dives.