%% Cell type:markdown id: tags:

AUV *Abyss* workflow for processing a single survey

----

M. Klischies, J. Mohrmann, I.A. Yeo, A. Steinführer, M. Rothenbeck

Version 3.0

Aims:

1. Relative renavigation of a single survey using feature matching in MBES data.

2. Shift of a single survey relative to georeferenced data to facilitate merging of multiple surveys.

Prerequisites:

- Raw data format: RESON s7k-files

- Software installed:

 - MB-Systems

 - Python 3

 - GIS program

------

**Workflow overview**

- Define basic parameters (mission number, spacial resolution, datalists, etc.)

.s7k -> datalist.mb-1

- *mbset* (sets lever arm offsets & biases in .s7k.par-files)

- *mbprocess* (datalist.mb-1 + .s7k.par = p.mb88 files)

p.mb88 -> datalist_p.mb-1

- Grid data to check quality

- *mbnavadjust* (set up Abyss0XXX_navadj-project with datalist_p.mb-1 to **renavigate** the mission)

- *mbprocess* (datalist_p.mb-1 + p.mb88.na0 = pp.mb88 files)

pp.mb88 -> datalist_pp.mb-1

- Grid data to check quality

pp.mb88 -> datalist_pp.mb-1

- Grid data to check quality and to decide, if horizontal shift needs to be applied

- **GIS project**: determine **horizontal shift** to match grid and ship's bathymetry

- *mbset* (set horizontal shift in x- and y-direction in pp.mb88.par files)

- *mbprocess* (datalist_p.mb-1 + pp.mb88.par = ppp.mb88 files)

ppp.mb88 -> datalist_ppp.mb-1

- Grid data to check shift

datalist_pp.mb-1 and pp.mb88 **or** datalist_ppp.mb-1 and ppp.mb88

- Export and archive data

- Prepare merging project

%% Cell type:markdown id: tags:

## Mission 0XXX

*(enter dive number)*

Enter short narrative on cruise/purpose/dive/circumstances/obstacles here.

-----

### Definition of basic parameters

%% Cell type:code id: tags:

``` python

# enter dive number

dive = "0XXX"

device = "Abyss"

# set name of this jupyter notebook file

# if this file is not in your working directory, specify the whole file path

notebook = "/path/to/notebook/AUVAbyssWorkflow_singlesurvey.ipynb"

# enter spatial resolution in meter

sr = 3

# enter rootpath of raw-data

rootpath = "/path/to/rawdata/Abyss0XXX"

print (dive)

print (sr)

# define mbsystem internal file ending

file_ending = ".mb88"

# define names of datalists

datalistraw = "datalistraw_" + str(dive) + ".mb-1"

datalist = "datalist_" + str(dive) + ".mb-1"

datalistp = "datalistp_" + str(dive) + ".mb-1"

datalistpp = "datalistpp_" + str(dive) + ".mb-1"

datalistppp = "datalistppp_" + str(dive) + ".mb-1"

# define names of grids

grd1 = device + "_" + str(dive) + "_raw_" + str(sr) + "m"

grd1nav = device + "_" + str(dive) + "_raw_nav"

grd2arcAscii = device + "_" + str(dive) + "_navadj_" + str(sr) + "m_arcAscii"

grd2netCDF = device + "_" + str(dive) + "_navadj_" + str(sr) + "m_netCDF"

grd3 = device + "_" + str(dive) + "_navadj2_" + str(sr) + "m"

grd4 = device + "_" + str(dive) + "_navadj_shift_" + str(sr) + "m"

# define name of navadjust project (make sure this alings with what you enter in navadjust later)

navadjustProjectName = "Abyss0XXX_navadj"

```

%% Cell type:markdown id: tags:

## Navigation adjustment

Operator: *(enter name here)*

20yy-mm-dd

Files rejected for further processing: *(list according files below)*

- *.s7k

- *.s7k

----

### Preprocessing

%% Cell type:code id: tags:

``` python

%cd $rootpath

print (datalistraw)

# Create datalist of raw files

!mbm_makedatalist -O$datalistraw

```

%% Cell type:code id: tags:

``` python

!mbpreprocess --input=$datalistraw

```

%% Cell type:code id: tags:

``` python

%cd $rootpath

# Create datalist of preprocessed files

searchString = "*" + file_ending

!ls -1 $searchString > $datalist

```

%% Cell type:code id: tags:

``` python

%cd $rootpath

# Set lever arm offset

!mbset -I$datalistraw -PSONAROFFSETX:-0.079 -PSONAROFFSETY:0.196 -PSONAROFFSETZ:0.048 -PVRUOFFSETX:-0.4473 -PVRUOFFSETY:0.000 -PVRUOFFSETZ:-0.3395

```

%% Cell type:code id: tags:

``` python

%cd $rootpath

# Set bias

!mbset -I$datalistraw -PROLLBIAS:0.17 -PPITCHBIAS:1.32 -PHEADINGOFFSET:0.93

```

%% Cell type:code id: tags:

``` python

%cd $rootpath

# apply offsets and biases

!mbprocess -I$datalistraw -V

```

%% Cell type:markdown id: tags:

### Grid to check quality

%% Cell type:code id: tags:

``` python

%cd $rootpath

# Create datalist of preprocessed files

searchString = "*p" + file_ending

!ls -1 $searchString > $datalistp

```

%% Cell type:code id: tags:

``` python

%cd $rootpath

print (grd1)

# First grid and view with MB Systems

!mbm_plot -I$datalistp -A1/45 -G2 -PA1 -N -O$grd1

# set name of generated script

grd1exec = grd1 + ".cmd"

# execute script:

!./$grd1exec

```

%% Cell type:code id: tags:

``` python

%cd $rootpath

print (grd1nav)

# Plot navigation and view with MB Systems

!mbm_plot -I$datalistp -N -O$grd1nav

# set name of generated script

grd1exec = grd1nav + ".cmd"

# execute script

!./$grd1exec

```

%% Cell type:code id: tags:

``` python

%cd $rootpath

# Creating arcAscii grid of raw data to view in GIS project

!mbgrid -I$datalistp -A2 -G4 -E$sr/0.0! -O$grd1 -V

```

%% Cell type:markdown id: tags:

### Navigation Adjustment

**Steps using mbnavadjust**

Run mbnavadjust command to open the graphical interface, then follow:

- File > New > Name navadjust project. (make sure this alings with what you define for the variable "navadjustProjectName")

- File > Import data > Choose datalist (datalistp_0XXX.mb-1).

- View > True Crossings > Double click on one file of the list > opens NavErr window.

- Adjust contour lines manually > Add Tie > Next ... and repeat. Avoid 'Minimum Missfit' and skip tiles with poor data. Dismiss, if you are done.

- View > >50% overlap > double click on one file of the list > opens NavErr window.

- Adjust contour lines manually > Add Tie > Next ... and repeat. Dismiss.

- Action > Invert.

- Action > Apply Adjusted Navigation.

- File > Quit.

%% Cell type:code id: tags:

``` python

print(navadjustProjectName)

```

%% Cell type:code id: tags:

``` python

%cd $rootpath

# First manual navadjusting: on true crossings and >50% overlap.

!mbnavadjust

```

%% Cell type:code id: tags:

``` python

%cd $rootpath

# Apply adjusted navigation

!mbprocess -I$datalistp -V

```

%% Cell type:code id: tags:

``` python

%cd $rootpath

searchString_pp = "*pp" + file_ending

!ls -1 $searchString_pp > $datalistpp

```

%% Cell type:markdown id: tags:

### Grid to check quality

%% Cell type:code id: tags:

``` python

%cd $rootpath

# Plot navigation track after navadjust

!mbm_plot -I$datalistpp -N -O$navadjustProjectName

# set name of generated script

navadjustProjectNameExec = navadjustProjectName + ".cmd"

!./$navadjustProjectNameExec

```

%% Cell type:code id: tags:

``` python

%cd $rootpath

# Creating arcAscii grid of navadjusted data

!mbgrid -I$datalistpp -A2 -G4 -E3/0.0! -O$grd2arcAscii

```

%% Cell type:code id: tags:

``` python

%cd $rootpath

# Creating netCDF grid of navadjusted data

!mbgrid -I$datalistpp -A2 -G2 -E3/0.0! -O$grd2netCDF

```

%% Cell type:markdown id: tags:

**Quality check of the adjusted navigation**

 - Navigation track is plausible: Continue with the next processing step (shift).

 - Still a lot of outliers in the grid: Maybe adjusting 25% overlaps will solve that.

**Further navigation adjustment**

- File > Open > Choose project and project file.

- View > >25% overlap > Double click on one file of the list > opens NavErr window.

- Adjust contour lines manually > Add Tie > Next ... and repeat. Dismiss.

- Action > Invert.

- Action > Apply Adjusted Navigation.

- File > Quit.

%% Cell type:code id: tags:

``` python

%cd $rootpath

# Reload project and add ties at 25% overlap.

!mbnavadjust

```

%% Cell type:code id: tags:

``` python

%cd $rootpath

!mbprocess -I$datalistp -V

```

%% Cell type:code id: tags:

``` python

%cd $rootpath

# Creating Ascii grid with the adjusted navigation based on the 25% overlap

!mbgrid -I$datalistpp -A2 -G4 -E3/0.0! -O$grd3

```

%% Cell type:code id: tags:

``` python

%cd $rootpath

# set names for second navadjust plot

navadjustPlot2 = navadjustProjectName + "2"

navadjustPlot2exec = navadjustPlot2 + ".cmd"

# Check navigation track

!mbm_plot -I$datalistpp -N -O$navadjustPlot2

!./$navadjustPlot2exec

```

%% Cell type:markdown id: tags:

**Result**

Number of set ties after analyzing >50% overlap crossings: *(enter number here)*

Number of set ties after analyzing >25% overlap crossings: *(enter number here)*

Enter narrative on obstacles and how they were overcome etc. here.

%% Cell type:markdown id: tags:

-----

## Shift renavigated data

*Operator: (enter name here)*

20yy-mm-dd

Horizontal shift of the data provides absolute spatial constrains, as it is applied relative to known, spatially referenced features, e.g. from ship-based bathymetry or seafloor video dives. With this, it facilitates the merging of multi-survey mapping campaigns. Shift only, when the accuracy of the spatial reference exceeds that of the initial AUV navigation (LBL network, if used).

**Determine the horizontal shift in a GIS project**

Shift grid (datalistpp) to match georeferenced data and enter distance and bearing below.

%% Cell type:code id: tags:

``` python

# enter bearing and distance of shift according to shift of ascii-grid in Global Mapper

dist=40     # distance in m

bear=260     # bearing in degree

import numpy as np

x_shift=np.sin(np.radians(bear))*dist

y_shift=np.cos(np.radians(bear))*dist

print (x_shift)

print (y_shift)

```

%% Cell type:code id: tags:

``` python

%cd $rootpath

# set offset in parameter file of navadjusted data

!mbset -I$datalistpp -PNAVSHIFTX:$x_shift -PNAVSHIFTY:$y_shift

```

%% Cell type:code id: tags:

``` python

%cd $rootpath

# Apply offset

!mbprocess -I$datalistpp -V

```

%% Cell type:code id: tags:

``` python

# Create new datalist --> neccessary for merging project

%cd $rootpath

searchString_ppp = "*ppp" + file_ending

!ls -1 $searchString_ppp > $datalistppp

#set name for plot

navadjustPlotShift = navadjustProjectName + "_shift"

navadjustPlotShiftExec = navadjustPlotShift + ".cmd"

# Plot navigation track after shift

!mbm_plot -I$datalistppp -N -O$navadjustPlotShift

!./$navadjustPlotShiftExec

# Creating Ascii grid of adjusted and shifted data to check shift

!mbgrid -I$datalistppp -A2 -G4 -E3/0.0! -O$grd4

```

%% Cell type:markdown id: tags:

## Exporting

Export to folder (this might take a while)

- the latest processed data (ppp-files with applied shift!)

- the latest grids

- the navadjust project

%% Cell type:code id: tags:

``` python

%cd $rootpath

#define export folder path

exportpath = rootpath + "/export"

#create export folder

!mkdir $exportpath

#export ppp files

searchString_ppp = "*ppp" + file_ending

!ls -1 $searchString_ppp | xargs cp -t $exportpath

#export grids

grd1File = grd1 + ".asc"

!cp $grd1File $exportpath

grd2File = grd2arcAscii + ".asc"

!cp $grd2File $exportpath

grd3File = grd3 + ".asc"

!cp $grd3File $exportpath

grd4File = grd4 + ".asc"

!cp $grd4File $exportpath

#exports navadjust project

#folder .dir

dirName = navadjustProjectName + ".dir"

!cp -a $dirName $exportpath

#file .nvh

nvhName = navadjustProjectName + ".nvh"

!cp $nvhName $exportpath

#export lists

mblist_name = exportpath + "/" + device + "_" + dive + "_renav-soundings.xyz"

mbnavlist_name = exportpath + "/" + device + "_" + dive + "_renav-vehicletrack.xyz"

mbnavlist_time_name = exportpath + "/" + device + "_" + dive + "_renav-vehicletrack_time.xyz"

!mblist -Idatalist.mb-1 -D3 > $mblist_name

!mbnavlist -Idatalist.mb-1 -OXYc > $mbnavlist_name

!mbnavlist -Idatalist.mb-1 -OJXYc > $mbnavlist_time_name ## for editing in Qimera

#save jupyter notebook

!cp $notebook $exportpath

```

%% Cell type:markdown id: tags:

----

## Merge Multi-Surveys

**Prepare merging project:**

- Use _pp.mb-1 and according files, if no shifting was applied.

- Use _ppp.mb-1 if shift was applied.

- Move the datalists and according files of all surveys to be merged into one common folder.

- Continue with

**AUVAbyssWorkflow_merge-multiplesurveys.ipynb** placed in that commmon folder

%% Cell type:code id: tags:

``` python

```