Merge remote-tracking branch 'origin/master'

%% Cell type:markdown id: tags:

AUV *Abyss* workflow for processing a single survey

----

M. Klischies, J. Mohrmann, I.A. Yeo, A. Steinführer, M. Rothenbeck

Version 3.0

Aims:

1. Relative renavigation of a single survey using feature matching in MBES data.

2. Shift of a single survey relative to georeferenced data to facilitate merging of multiple surveys.

Prerequisites:

- Raw data format: RESON s7k-files

- Software installed:

 - MB-Systems

 - Python 3

 - GIS program

------

**Workflow overview**

- Define basic parameters (dive number, spacial resolution, datalists, etc.)

.s7k -> datalist.mb-1

- *mbset* (sets lever arm offsets & biases in .s7k.par-files)

- *mbprocess* (datalist.mb-1 + .s7k.par = p.mb88 files)

p.mb88 -> datalist_p.mb-1

- Grid data to check quality

- *mbnavadjust* (set up Abyss0XXX_navadj-project with datalist_p.mb-1 to **renavigate** the dive)

- *mbprocess* (datalist_p.mb-1 + p.mb88.na0 = pp.mb88 files)

pp.mb88 -> datalist_pp.mb-1

- Grid data to check quality

pp.mb88 -> datalist_pp.mb-1

- Grid data to check quality and to decide, if horizontal shift needs to be applied

- **GIS project**: determine **horizontal shift** to match grid and ship's bathymetry

- *mbset* (set horizontal shift in x- and y-direction in pp.mb88.par files)

- *mbprocess* (datalist_p.mb-1 + pp.mb88.par = ppp.mb88 files)

ppp.mb88 -> datalist_ppp.mb-1

- Grid data to check shift

datalist_pp.mb-1 and pp.mb88 **or** datalist_ppp.mb-1 and ppp.mb88

- Export and archive data

- Prepare merging project

%% Cell type:markdown id: tags:

## Dive 0XXX

*(enter dive number)*

Enter short narrative on cruise/purpose/dive/circumstances/obstacles here.

-----

### Definition of basic parameters

%% Cell type:code id: tags:

``` python

# enter dive number

dive = "0XXX"

device = "Abyss"

# set name of this jupyter notebook file

# if this file is not in your working directory, specify the whole file path

notebook = "/path/to/notebook/AUVAbyssWorkflow_singlesurvey_MK.ipynb"

# enter spatial resolution in meter

sr = 3

# enter rootpath of raw-data

rootpath = "/media/jochen/Work/AUV_navadjust_4Jochen/Abyss0253"

rootpath = "path/to/data/Abyss0253"

print (dive)

print (sr)

# define mbsystem internal file ending

file_ending = ".mb88"

# define names of datalists

datalistraw = "datalistraw_" + str(dive) + ".mb-1"

datalistp = "datalistp_" + str(dive) + ".mb-1"

datalistpp = "datalistpp_" + str(dive) + ".mb-1"

datalistppp = "datalistppp_" + str(dive) + ".mb-1"

# define names of grids

grd1 = "Abyss_" + str(dive) + "_raw_" + str(sr) + "m"

grd1nav = "Abyss_" + str(dive) + "_raw_nav"

grd2arcAscii = "Abyss_" + str(dive) + "_navadj_" + str(sr) + "m_arcAscii"

grd2netCDF = "Abyss_" + str(dive) + "_navadj_" + str(sr) + "m_netCDF"

grd3 = "Abyss" + str(dive) + "_navadj2_" + str(sr) + "m"

grd4 = "Abyss" + str(dive) + "_navadj_shift_" + str(sr) + "m"

grd1 = device + "_" + str(dive) + "_raw_" + str(sr) + "m"

grd1nav = device + "_" + str(dive) + "_raw_nav"

grd2arcAscii = device + "_" + str(dive) + "_navadj_" + str(sr) + "m_arcAscii"

grd2netCDF = device + "_" + str(dive) + "_navadj_" + str(sr) + "m_netCDF"

grd3 = device + "_" + str(dive) + "_navadj2_" + str(sr) + "m"

grd4 = device + "_" + str(dive) + "_navadj_shift_" + str(sr) + "m"

# define name of navadjust project (make sure this alings with what you enter in navadjust later)

navadjustProjectName = "Abyss0250_navadj"

```

%% Output

    0XXX

    3

%% Cell type:markdown id: tags:

## Renavigation

Operator: *(enter name here)*

20yy-mm-dd

Files rejected for further processing: *(list according files below)*

- *.s7k

- *.s7k

----

### Preprocessing

%% Cell type:code id: tags:

``` python

%cd $rootpath

print (datalistraw)

# Create datalist of raw files

!mbm_makedatalist -O$datalistraw

```

%% Output

    datalistraw_0XXX.mb-1

    'mbm_makedatalist' is not recognized as an internal or external command,

    operable program or batch file.

%% Cell type:code id: tags:

``` python

%cd $rootpath

# Set lever arm offset

!mbset -I$datalistraw -PSONAROFFSETX:-0.079 -PSONAROFFSETY:0.196 -PSONAROFFSETZ:0.048 -PVRUOFFSETX:-0.4473 -PVRUOFFSETY:0.000 -PVRUOFFSETZ:-0.3395

```

%% Cell type:code id: tags:

``` python

%cd $rootpath

# Set bias

!mbset -I$datalistraw -PROLLBIAS:0.17 -PPITCHBIAS:1.32 -PHEADINGOFFSET:0.93

```

%% Cell type:code id: tags:

``` python

%cd $rootpath

# apply offsets and biases

!mbprocess -I$datalistraw -V

```

%% Cell type:markdown id: tags:

### Grid to check quality

%% Cell type:code id: tags:

``` python

%cd $rootpath

print (datalistp)

# Create datalist of preprocessed files

!ls -1 *p.mb88 > $datalistp

searchString = "*p" + file_ending

!ls -1 $searchString > $datalistp

```

%% Output

    /media/jochen/Work/AUV_navadjust_4Jochen/Abyss0253

    datalistp_0XXX.mb-1

    20161214_173622p.mb88

    20161214_173622pp.mb88

    20161214_175151p.mb88

    20161214_175151pp.mb88

    20161214_180749p.mb88

    20161214_180749pp.mb88

    20161214_182409p.mb88

    20161214_182409pp.mb88

    20161214_184018p.mb88

    20161214_184018pp.mb88

    20161214_185640p.mb88

    20161214_185640pp.mb88

    20161214_191237p.mb88

    20161214_191237pp.mb88

    20161214_192807p.mb88

    20161214_192807pp.mb88

    20161214_194327p.mb88

    20161214_194327pp.mb88

    20161214_195900p.mb88

    20161214_195900pp.mb88

    20161214_201443p.mb88

    20161214_201443pp.mb88

    20161214_203113p.mb88

    20161214_203113pp.mb88

    20161214_204754p.mb88

    20161214_204754pp.mb88

    20161214_210432p.mb88

    20161214_210432pp.mb88

    20161214_212047p.mb88

    20161214_212047pp.mb88

    20161214_213715p.mb88

    20161214_213715pp.mb88

    20161214_215350p.mb88

    20161214_215350pp.mb88

    20161214_221009p.mb88

    20161214_221009pp.mb88

    20161214_222633p.mb88

    20161214_222633pp.mb88

    20161214_224242p.mb88

    20161214_224242pp.mb88

    20161214_225915p.mb88

    20161214_225915pp.mb88

    20161214_231534p.mb88

    20161214_231534pp.mb88

    20161214_233155p.mb88

    20161214_233155pp.mb88

    20161214_234754p.mb88

    20161214_234754pp.mb88

    20161215_000427p.mb88

    20161215_000427pp.mb88

    20161215_002123p.mb88

    20161215_002123pp.mb88

    20161215_003810p.mb88

    20161215_003810pp.mb88

    20161215_005518p.mb88

    20161215_005518pp.mb88

%% Cell type:code id: tags:

``` python

%cd $rootpath

print (grd1)

# First grid and view with MB Systems

!mbm_plot -I$datalistp -A1/45 -G2 -PA1 -N -O$grd1

# set name of generated script

grd1exec = grd1 + ".cmd"

# execute script:

!./$grd1exec

```

%% Cell type:code id: tags:

``` python

%cd $rootpath

print (grd1nav)

# Plot navigation and view with MB Systems

!mbm_plot -I$datalistp -N -O$grd1nav

# set name of generated script

grd1exec = grd1nav + ".cmd"

# execute script

!./$grd1exec

```

%% Cell type:code id: tags:

``` python

%cd $rootpath

# Creating arcAscii grid of raw data to view in Global Mapper

!mbgrid -I$datalistp -A2 -G4 -E$sr/0.0! -O$grd1 -V

```

%% Cell type:markdown id: tags:

### Navigation Adjustment

**Steps using mbnavadjust**

Run mbnavadjust command to open the graphical interface, then follow:

- File > New > Name navadjust project. (make sure this alings with what you define for the variable "navadjustProjectName")

- File > Import data > Choose datalist (datalistp_0XXX.mb-1).

- View > True Crossings > Double click on one file of the list > opens NavErr window.

- Adjust contour lines manually > Add Tie > Next ... and repeat. Avoid 'Minimum Missfit' (what algorithm is behind this? => Cross Correlation). Dismiss, if you are done.

- View > >50% overlap > double click on one file of the list > opens NavErr window.

- Adjust contour lines manually > Add Tie > Next ... and repeat. Dismiss.

- Action > Invert.

- Action > Apply Adjusted Navigation.

- File > Quit.

%% Cell type:code id: tags:

``` python

print(navadjustProjectName)

```

%% Cell type:code id: tags:

``` python

%cd $rootpath

# First manual navadjusting: on true crossings and >50% overlap.

!mbnavadjust

```

%% Cell type:code id: tags:

``` python

%cd $rootpath

# Apply adjusted navigation

!mbprocess -I$datalistp -V

```

%% Cell type:code id: tags:

``` python

%cd $rootpath

!ls -1 *pp.mb88 > $datalistpp

searchString_pp = "*pp" + file_ending

!ls -1 $searchString_pp > $datalistpp

```

%% Cell type:markdown id: tags:

### Grid to check quality

%% Cell type:code id: tags:

``` python

%cd $rootpath

# Plot navigation track after navadjust

!mbm_plot -I$datalistpp -N -O$navadjustProjectName

# set name of generated script

navadjustProjectNameExec = navadjustProjectName + ".cmd"

!./$navadjustProjectNameExec

```

%% Cell type:code id: tags:

``` python

%cd $rootpath

# Creating arcAscii grid of navadjusted data

!mbgrid -I$datalistpp -A2 -G4 -E3/0.0! -O$grd2arcAscii

```

%% Cell type:code id: tags:

``` python

%cd $rootpath

# Creating netCDF grid of navadjusted data

!mbgrid -I$datalistpp -A2 -G2 -E3/0.0! -O$grd2netCDF

```

%% Cell type:markdown id: tags:

**Quality check of the adjusted navigation**

 - Navigation track is plausible: Continue with the next processing step (shift).

 - Still a lot of outliers in the grid: Maybe adjusting 25% overlaps will solve that.

**Further navigation adjustment**

- File > Open > Choose project and project file.

- View > >25% overlap > Double click on one file of the list > opens NavErr window.

- Adjust contour lines manually > Add Tie > Next ... and repeat. Dismiss.

- Action > Invert.

- Action > Apply Adjusted Navigation.

- File > Quit.

%% Cell type:code id: tags:

``` python

%cd $rootpath

# Reload project and add ties at 25% overlap.

!mbnavadjust

```

%% Cell type:code id: tags:

``` python

%cd $rootpath

!mbprocess -I$datalistp -V

```

%% Cell type:code id: tags:

``` python

%cd $rootpath

# Creating Ascii grid with the adjusted navigation based on the 25% overlap

!mbgrid -I$datalistpp -A2 -G4 -E3/0.0! -O$grd3

```

%% Output

    /media/jochen/Work/AUV_navadjust_4Jochen/Abyss0253

    Unable to open data list file: datalistpp_0XXX.mb-1

    Program terminated in function <mb_get_info_datalist>

%% Cell type:code id: tags:

``` python

%cd $rootpath

# set names for second navadjust plot

navadjustPlot2 = navadjustProjectName + "2"

navadjustPlot2exec = navadjustPlot2 + ".cmd"

# Check navigation track

!mbm_plot -I$datalistpp -N -O$navadjustPlot2

!./$navadjustPlot2exec

```

%% Cell type:markdown id: tags:

**Result**

Number of set ties after analyzing >50% overlap crossings: *(enter number here)*

Number of set ties after analyzing >25% overlap crossings: *(enter number here)*

Enter narrative on obstacles and how they were overcome etc. here.

%% Cell type:markdown id: tags:

-----

## Shift renavigated data

*Operator: (enter name here)*

20yy-mm-dd

Horizontal shift of the data provides absolute spatial constrains, as it is applied relative to known, spatially referenced features, e.g. from ship-based bathymetry or seafloor video dives. With this, it facilitates the merging of multi-survey mapping campaigns.

**Determine the horizontal shift in a GIS project**

Shift grid (datalistpp) to match georeferenced data and enter distance and bearing below.

%% Cell type:code id: tags:

``` python

# enter bearing and distance of shift according to shift of ascii-grid in Global Mapper

dist=40     # distance in m

bear=260     # bearing in degree

import numpy as np

x_shift=np.sin(np.radians(bear))*dist

y_shift=np.cos(np.radians(bear))*dist

print (x_shift)

print (y_shift)

```

%% Cell type:code id: tags:

``` python

%cd $rootpath

# set offset in parameter file of navadjusted data

!mbset -I$datalistpp -PNAVSHIFTX:$x_shift -PNAVSHIFTY:$y_shift

```

%% Cell type:code id: tags:

``` python

%cd $rootpath

# Apply offset

!mbprocess -I$datalistpp -V

```

%% Cell type:code id: tags:

``` python

# Create new datalist --> neccessary for merging project

%cd $rootpath

!ls -1 *ppp.mb88 > $datalistppp

searchString_ppp = "*ppp" + file_ending

!ls -1 $searchString_ppp > $datalistppp

#set name for plot

navadjustPlotShift = navadjustProjectName + "_shift"

navadjustPlotShiftExec = navadjustPlotShift + ".cmd"

# Plot navigation track after shift

!mbm_plot -I$datalistppp -N -O$navadjustPlotShift

!./$navadjustPlotShiftExec

# Creating Ascii grid of adjusted and shifted data to check shift

!mbgrid -I$datalistppp -A2 -G4 -E3/0.0! -O$grd4

```

%% Cell type:markdown id: tags:

## Exporting

Export to folder (this might take a while)

- the latest processed data (ppp-files with applied shift!)

- the latest grids

- the navadjust project

%% Cell type:code id: tags:

``` python

%cd $rootpath

!pwd

#define export folder path

exportpath = rootpath + "/export"

#create export folder

!mkdir $exportpath

#export ppp files

!ls -1 *ppp.mb88 | xargs cp -t $exportpath

searchString_ppp = "*ppp" + file_ending

!ls -1 $searchString_ppp | xargs cp -t $exportpath

#export grids

grd1File = grd1 + ".asc"

!cp $grd1File $exportpath

grd2File = grd2arcAscii + ".asc"

!cp $grd2File $exportpath

grd3File = grd3 + ".asc"

!cp $grd3File $exportpath

grd4File = grd4 + ".asc"

!cp $grd4File $exportpath

#exports navadjust project

#folder .dir

dirName = navadjustProjectName + ".dir"

!cp -a $dirName $exportpath

#file .nvh

nvhName = navadjustProjectName + ".nvh"

!cp $nvhName $exportpath

#export lists

mblist_name = exportpath + "/" + device + "_" + dive + "_renav-soundings.xyz"

mbnavlist_name = exportpath + "/" + device + "_" + dive + "_renav-vehicletrack.xyz"

mbnavlist_time_name = exportpath + "/" + device + "_" + dive + "_renav-vehicletrack_time.xyz"

!mblist -Idatalist.mb-1 -D3 > $mblist_name

!mbnavlist -Idatalist.mb-1 -OXYc > $mbnavlist_name

!mbnavlist -Idatalist.mb-1 -OJXYc > $mbnavlist_time_name ## for editing in Qimera

#save jupyter notebook

!cp $notebook $exportpath

```

%% Output

    /media/jochen/Work/AUV_navadjust_4Jochen/Abyss0253

    /media/jochen/Work/AUV_navadjust_4Jochen/Abyss0253

    mkdir: cannot create directory ‘/media/jochen/Work/AUV_navadjust_4Jochen/Abyss0253/export’: File exists

%% Cell type:code id: tags:

``` python

!mblist -Idatalist.mb-1 -D3 > Abyss0XXX_renav-soundings.xyz

```

%% Cell type:code id: tags:

``` python

!mbnavlist -Idatalist.mb-1 -OXYc >> Abyss0XXX_renav-vehicletrack.xyz

!mbnavlist -Idatalist.mb-1 -OJXYc >> Abyss0XXX_renav-vehicletrack+time.xyz ## for editing in Qimera

```

    ls: cannot access '*ppp.mb88': No such file or directory

    cp: missing file operand

    Try 'cp --help' for more information.

    cp: cannot stat 'Abyss_0XXX_raw_3m.asc': No such file or directory

    cp: cannot stat 'Abyss_0XXX_navadj_3m_arcAscii.asc': No such file or directory

    cp: cannot stat 'Abyss_0XXX_navadj2_3m.asc': No such file or directory

    cp: cannot stat 'Abyss_0XXX_navadj_shift_3m.asc': No such file or directory

    cp: cannot stat 'Abyss0250_navadj.dir': No such file or directory

    cp: cannot stat 'Abyss0250_navadj.nvh': No such file or directory

    Unable to open data list file: datalist.mb-1

    Program <MBLIST> Terminated

    Unable to open data list file: datalist.mb-1

    Program <mbnavlist> Terminated

    Unable to open data list file: datalist.mb-1

    Program <mbnavlist> Terminated

%% Cell type:markdown id: tags:

----

## Merge Multi-Surveys

**Prepare merging project:**

- Use _pp.mb-1 and according files, if no shifting was applied.

- Use _ppp.mb-1 if shift was applied.

- Move the datalists and according files of all surveys to be merged into one common folder.

- Continue with

**AUVAbyssWorkflow_merge-multiplesurveys.ipynb** placed in that commmon folder

%% Cell type:code id: tags:

``` python

```