created merge prototype

%% Cell type:markdown id: tags:

AUV Abyss workflow for merging of multi-surveys

------

M. Klischies, J. Mohrmann, I.A. Yeo, A. Steinführer, M. Rothenbeck

Workflow Version 2.0

Aim:

Merge MBES data of a multi-survey mapping campaign to produce a consistent data set and a large-coverage map.

-----

**Workflow overview**

Depending on the post-processing treatment of the individual surveys, the merging workflow branches into two different approaches.

**A) Applies to renavigated and shifted data.** *Preferred option.*

- Prepare merge project folder

ppp.mb88 and datalist_ppp.mb-1

- *mbnavadjust*

 - Set up **merge project**

 - Add data of two missions

 - Fix navigation of each individual mission

 - **Navigation adjustment in overlapping areas**

- *mbprocess* (_ppp.mb-1 datalists + ppp.mb88.na0 = pppp.mb88 files)

pppp.mb88 -> datalist_pppp.mb-1

- Grid data to check quality

ppp.mb88 and datalist_ppp.mb-1

- *mbnavadjust* (**Reopen** merge project)

 - Add another mission

 - Fix its navigation

 - Renavigate overlapping areas with the other dives

- Repeat running *mbprocess* (_ppp.mb-1 datalists + ppp.mb88.na0 = pppp.mb88 files) and gridding data (pppp.mb88 -> datalist_pppp.mb-1) to check quality

- Reopen merge project and add another dives. Consecutively continue until all surveys are merged.

- *mbprocess* (_ppp.mb-1 datalists + ppp.mb88.na0 = pppp.mb88 files)

pppp.mb88 -> datalist_pppp.mb-1

- Grid data to check quality

- Export of navigation adjusted, shifted and merged data

**B) Applies to renavigated, but unshifted data.**

pp.mb88 and datalist_pp.mb-1

- *mbnavadjustmerge* (combines existing navadjust projects and sets up an overall **merge project**)

- *mbnavadjust* (choose the created merge project to **adjust navigation in overlapping areas** and merge the surveys)

- *mbprocess* (_pp.mb-1 datalists + pp.mb88.na0 = ppp.mb88 files)

ppp.mb88 -> datalist_ppp.mb-1

- Grid data to check quality

- Export navigation adjusted, merged data

%% Cell type:markdown id: tags:

## Target area

*(enter mapped area)*

Missions to be merged: *(list according surveys below)*

- 0XXX1

- 0XXX2 ...

Enter short narrative on cruise/purpose/dive/circumstances/obstacles here.

Absolute navigation from: *(list georeferenced features with dive numbers below)*

- ...

- *(e.g. IODP## re-entry cone visible in dive 0XXX and 0XXX; fault scarp in dive 0XXX and 0XXX from ROV sampling)*

------

## Define basic parameters

%% Cell type:code id: tags:

``` python

rootpath="/enter/path/here"

%cd $rootpath

# define name of navadjust project (make sure this alings with what you enter in navadjust later)

navadjustProjectName = "targetarea"

# datalists

datalist_navadjust = navadjustProjectName + ".mb-1"

file_ending = ".mb88"

```

%% Output

    [Errno 2] No such file or directory: '/enter/path/here'

    /home/jochen/workspace/MBSystemWorkflow/AUVAbyssWorkflow

%% Cell type:markdown id: tags:

## Prepare merge project

Create a merge project folder containing the following:

A)

- this jupyter notebook

- a copy of the dive files to be merged (_ppp.mb88 files)

- a copy of the corresponding datalists (_ppp.mb-1 datalists)

B)

- this jupyter notebook

- a copy of the dive files to be merged (_pp.mb88 files)

- a copy of the corresponding datalists (_pp.mb-1 datalists)

- a copy of the existing, corresponding navadjust project files

 - Abyss0XXX_navadj.mb-1

 - Abyss0XXX_navadj.dir folder

 - Abyss0XXX_navadj.nvh

 - Abyss0XXX_navadj_dx.txt

 - Abyss0XXX_navadj_dy.txt

 - *.rte files

-----

## Merge project

A) Data of each mission is shifted in respect to the above mentioned absolute navigation values, and/or referenced ship bathymetry (note the accuracies of available absolute navigation fixes!). So, each survey should be in the approximate right location. Disadvantage: set ties of previous navadjust-projects are lost. Therefore: **Prior to adjust overlapping areas, set ties within each survey to keep the previously adjusted navigation!**

B) Use the tool *mbnavadjustmerge* to merge existing, not shifted navadjust-projects. Advantage: set ties of previous projects are translated. Disadvantage: only relative navigation constrains. **Limited to maximum offset of 400m between merged surveys!**

-----

%% Cell type:markdown id: tags:

### Merge project - approach A)

#### Mission 0XXX1 + 0XXX2

*Enter short narrative of dive areas, overlap, and available navigation references.*

**Set up a new mbnavadjust project**

- File > New > Name navadjust project: navadjustProjectName

- File > Import Swath Data > Choose datalistdive0XXX1

- File > Import Swath Data > Choose datalistdive0XXX2

1) Set ties to keep internal navigation of each survey:

- View > Show Survey-vs-Survey Blocks > Choose Block survey 0 vs. survey 0 (subsequently 1 vs 1)

- View > Show Only Selected Survey-vs-Survey Block

- View > Show True Crossings > navadjust in NavErr window. Dismiss.

- View > Show Crossings with 50% overlap > navadjust in NavErr window. Dismiss.

- Action > Invert

2) Merge overlapping areas:

- View > Show Survey-vs-Survey Blocks > Choose Block with survey0 vs survey1

- View > Show Only Selected Survey-vs-Survey Block

- View > Show True Crossings > navadjust in NavErr window. **Set ties carefully!** Dismiss. Repeat itinerary on more detailled levels, if neccessary.

- Action > Invert

- Action > Apply Adjusted Navigation.

- File > Quit.

3) Check Quality, adjust, continue with adding a next mission.

%% Cell type:markdown id: tags:

### Merge Prototype

%% Cell type:code id: tags:

``` python

print(navadjustProjectName)

```

%% Cell type:code id: tags:

``` python

#define merge variables

mission_a = "0000"

mission_b = "0001"

processing_level = "pppp" #after processing navadjusted files

datalistmerged = "datalistmerged_" + mission_a + "-" + mission_b + "_" + processing_level + ".mb-1"

plotname = navadjustProjectName + "_" + mission_a + "-" + mission_b + "_merge-nav"

gridname = navadjustProjectName + "_" + mission_a + "-" + mission_b + "_merge_3m"

```

%% Cell type:code id: tags:

``` python

%cd $rootpath

!mbnavadjust

```

%% Cell type:code id: tags:

``` python

%cd $rootpath

# process data using datalist automatically created by mbnavadjust

!mbprocess -I$datalist_navadjust -V

```

%% Cell type:code id: tags:

``` python

%cd $rootpath

# generate new datalist of processed files

searchString = "*" + processing_level + file_ending

!ls -1 $searchString > $datalistmerged

```

%% Cell type:code id: tags:

``` python

%cd $rootpath

# Plot navigation

!mbm_plot -I$datalistmerged -N -O$plotname

execName = plotnames + ".cmd"

!./$execName

```

%% Cell type:code id: tags:

``` python

%cd $rootpath

# Creating arcAscii grid of merged surveys

!mbgrid -I$datalistmerged -A2 -G4 -E3/0.0! -O$gridname

```

%% Cell type:markdown id: tags:

#### Example

%% Cell type:code id: tags:

``` python

print(navadjustProjectName)

```

%% Cell type:code id: tags:

``` python

!mbnavadjust

```

%% Cell type:code id: tags:

``` python

# process data using datalist automatically created by mbnavadjust

!mbprocess -IMerge_Irinovskoie.mb-1 -V

```

%% Cell type:code id: tags:

``` python

# generate new datalist of processed files

!ls -1 *pppp.mb88 > datalistmerged_0142-0143_pppp.mb-1

```

%% Cell type:code id: tags:

``` python

# Plot navigation

!mbm_plot -Idatalistmerged_0142-0143_pppp.mb-1 -N -OAbyss_0142-0143_merge-nav

!./Abyss_0142-0143_merge-nav.cmd

```

%% Cell type:code id: tags:

``` python

# Creating arcAscii grid of merged surveys

!mbgrid -Idatalistmerged_0142-0143_pppp.mb-1 -A2 -G4 -E3/0.0! -OAbyss_0142-0143_merge_3m

```

%% Cell type:markdown id: tags:

**Quality check of the adjusted navigation**

 - Navigation track is plausible and merged grid appears good: Continue with adding the next mission.

 - Merged areas appear chaotic/erroneous: Carefully set some more ties.

 - Renavigation causes a missfit in the survey, for which absolute navigation constrains are available: Set navigation of this survey as **good** and for the other ones as **poor**. Invert and apply navigation.

**Intermediate result**

*Enter narrative comment about navigation adjustment progress here.*

#### Mission 0XXX3

*Enter short narrative of dive areas, overlap, and available navigation references.*

**Adding data to existing mbnavadjust-project**

- Re-open mbnavajust project

- File > Import Swath Data > Choose datalistdive0XXX3

1) Set ties to keep internal navigation of each survey:

- View > Show Survey-vs-Survey Blocks > Choose Block survey 2 vs. survey 2

- View > Show Only Selected Survey-vs-Survey Block

- View > Show True Crossings > navadjust in NavErr window. Dismiss.

- View > Show Crossings with 50% overlap > navadjust in NavErr window. Dismiss.

- Action > Invert

2) Merge overlapping areas:

- View > Show Survey-vs-Survey Blocks > Choose Block with survey0 vs survey 2.

- View > Show Only Selected Survey-vs-Survey Block.

- View > Show True Crossings > navadjust in NavErr window. **Set ties carefully!** Dismiss.

- Repeat itinerary on survey1 vs survey2 block.

- Action > Invert

- Action > Apply Adjusted Navigation.

- File > Quit.

3) Check Quality, adjust, continue with adding a next mission.

%% Cell type:code id: tags:

``` python

!mbnavadjust

```

%% Cell type:code id: tags:

``` python

# process data using datalist automatically created by mbnavadjust

!mbprocess -IMerge_Irinovskoie.mb-1 -V

# new datalist for extended merging project

!ls -1 *pppp.mb88 > datalistmerged_0142-0143-0144_pppp.mb-1

```

%% Cell type:code id: tags:

``` python

# Plot navigation

!mbm_plot -Idatalistmerged_0142-0143-0144_pppp.mb-1 -N -OAbyss0242-to-0245_Cluster03_merge-nav

!./Abyss_0142-0143-0144_merge-nav.cmd

# Creating arcAscii grid of merged surveys

!mbgrid -Idatalistmerged_0142-0143-0144_pppp.mb-1 -A2 -G4 -E3/0.0! -OAbyss_0142-0143-0144_merge_3m

```

%% Cell type:markdown id: tags:

**Quality check of the adjusted navigation**

 - Navigation track is plausible and merged grid appears good: Continue with adding the next dive.

 - Merged areas appear chaotic/erroneous: Carefully set some more ties.

 - Renavigation causes a missfit in the survey, for which absolute navigation constrains are available: Set navigation of this survey as **good** and for the other ones as **poor**. Invert and apply navigation.

**Intermediate result**

*Enter narrative comment about navigation adjustment progress here.*

#### Mission 0XXX4

*Enter short narrative of dive areas, overlap, and available navigation references.*

**Adding data to existing mbnavadjust-project**

Continue as described above until all dives are added and merged.

%% Cell type:code id: tags:

``` python

!mbnavadjust

```

%% Cell type:code id: tags:

``` python

# process data using datalist automatically created by mbnavadjust

!mbprocess -IMerge_Irinovskoie.mb-1 -V

# new datalist for extended merging project

!ls -1 *pppp.mb88 >> datalistmerged_0141-0142-0143-0144_pppp.mb-1

```

%% Cell type:code id: tags:

``` python

# Plot navigation

!mbm_plot -Idatalistmerged_0141-0142-0143-0144_pppp.mb-1 -N -OAbyss_0141-0142-0143-0144_merge-nav

!./Abyss_0141-0142-0143-0144_merge-nav.cmd

# Creating arcAscii grid of merged surveys

!mbgrid -Idatalistmerged_0141-0142-0143-0144_pppp.mb-1 -A2 -G4 -E3/0.0! -OAbyss_0141-0142-0143-0144_merge_3m

```

%% Cell type:markdown id: tags:

**Quality check of the adjusted navigation**

 - Navigation track is plausible and merged grid appears good: Continue with adding the next dive.

 - Merged areas appear chaotic/erroneous: Carefully set some more ties.

 - Renavigation causes a missfit in the survey, for which absolute navigation constrains are available (Check in GIS project): Set navigation of this survey as **good** and for the other ones as **poor**, then invert and apply navigation (in mbnavadjust).

**Intermediate result**

*Enter narrative comment about navigation adjustment progress here.*

#### Dive 0XXX4

*Enter short narrative of dive areas, overlap, and available navigation references.*

%% Cell type:code id: tags:

``` python

!mbnavadjust

```

%% Cell type:code id: tags:

``` python

# process data using datalist automatically created by mbnavadjust

!mbprocess -IMerge_Irinovskoie.mb-1 -V

# new datalist for extended merging project

!ls -1 *pppp.mb88 >> datalistmerged_0140-to-0144_pppp.mb-1

```

%% Cell type:code id: tags:

``` python

# Plot navigation

!mbm_plot -Idatalistmerged_0140-to-0144_pppp.mb-1 -N -OAbyss_0140-to-0144_Cluster03_merge-nav

!./Abyss_0140-to-0144_merge-nav.cmd

# Creating arcAscii grid of merged surveys

!mbgrid -Idatalistmerged_0140-to-0144_pppp.mb-1 -A2 -G4 -E3/0.0! -OAbyss_0140-to-0144_merge_3m

```

%% Cell type:markdown id: tags:

### Merge project - approach B)

#### Mission 0XXX1 + 0XXX2

*Enter short narrative of dive areas, overlap, and available navigation references.*

**Setup a mbnavadjustmerge project**

1) Merge existing navadjust projects

- run mbnavadjustmerge command

2) Merge overlapping areas

- File > Open > Choose navadjustmerge project

- View > Show Survey-vs-Survey Blocks > Choose Block with survey0 vs survey1

- View > Show Only Selected Survey-vs-Survey Block

- View > Show True Crossings > navadjust in NavErr window. **Set ties carefully!** Dismiss. Repeat itinerary on more detailled levels, if neccessary.

- Action > Invert

- Action > Apply Adjusted Navigation.

- File > Quit.

3) Check Quality, adjust, continue with adding a next mission.

**Adding data to existing mbnavadjust-project**

 ***CAUTION:***

 *This workflow has NOT been tested for merging of more than two surveys!*

%% Cell type:code id: tags:

``` python

!mbnavadjustmerge --input=Abyss0XXX1-navadj1.nvh --input=Abyss0XXX2-navadj1.nvh --output=Abyss0XXX12_navadj1_merge.nvh

```

%% Cell type:code id: tags:

``` python

# process data using datalist automatically created by mbnavadjust

!mbprocess -IAbyss0XXX12_navadj1_merge.mb-1 -V

# new datalist for extended merging project

!ls -1 *ppp.mb88 >> datalistmerged_Abyss0XXX12_ppp.mb-1

```

%% Cell type:code id: tags:

``` python

# Plot navigation

!mbm_plot -Idatalistmerged_Abyss0XXX12_ppp.mb-1 -N -OAbyss0XXX12_merge-nav

!./Abyss_0140-to-0144_merge-nav.cmd

# Creating arcAscii grid of merged surveys

!mbgrid -Idatalistmerged_Abyss0XXX12_ppp.mb-1 -A2 -G4 -E3/0.0! -OAbyss0XXX12_merge_3m

```

%% Cell type:markdown id: tags:

**Result**

*Enter narrative comment about navigation adjustment progress here.*

%% Cell type:markdown id: tags:

--------

### Result of the merge project

Appraoch A/B was applied, because ...

The merged data is ...

*Enter narrative on the quality of the navigation adjustment here.*

-----

## Shift merged dataset

*Operator: (enter name here)*

20yy-mm-dd

Horizontal shift is applied to the entire data set to match other, georeferenced locations or data.

**Determine the horizontal shift in a GIS project**

Shift grid (datalistpppp) to match georeferenced data; enter distance and bearing below.

%% Cell type:code id: tags:

``` python

# enter bearing and distance of shift according to shift of ascii-grid in GIS

dist=25     # distance in m

bear=10     # bearing in degree

import numpy as np

x_shift=np.sin(np.radians(bear))*dist

y_shift=np.cos(np.radians(bear))*dist

print (x_shift)

print (y_shift)

```

%% Cell type:code id: tags:

``` python

# set offset in parameter file of merged data

!mbset -Idatalistmerged_0140-to-0144_pppp.mb-1 -PNAVSHIFTX:$x_shift -PNAVSHIFTY:$y_shift

```

%% Cell type:code id: tags:

``` python

# apply shift to merged data

!mbprocess -Idatalistmerged_0140-to-0144_pppp.mb-1 -V

```

%% Cell type:code id: tags:

``` python

# Create new datalist --> neccessary for merging project

!cd $rootpath

!ls -1 *ppppp.mb88 > datalistmergedshifted_0140-to-0144_ppppp.mb-1

# Plot navigation track after shift

!mbm_plot -Idatalistmergedshifted_0140-to-0144_ppppp.mb-1 -N -OAbyss0140-0144_Irinovskoie_mergedshift

!./Abyss0140-0144_Irinovskoie_mergedshift.cmd

# Creating Ascii grid of adjusted and shifted data to check shift

!mbgrid -Idatalistmergedshifted_0140-to-0144_ppppp.mb-1 -A2 -G4 -E3/0.0! -OAbyss0140-0144_Irinovskoie_mergedshift_3m

```

%% Cell type:markdown id: tags:

### Final result

*Enter narrative on the final result of the navigation adjustment here.*

-------

## Converting .mb88 Files into .xyz-data

To clean the navigation adjusted, shifted data, the sounding information must be exported to a file format readable by common MBES processing software, such as Qimera, or CARIS HIPS.

%% Cell type:code id: tags:

``` python

!mblist -Idatalistmergedshifted_0140-to-0144_ppppp.mb-1 -D3 > ODEMAR_Abyss_0140-0144_Irinovskoe_merged.xyz

```

%% Cell type:code id: tags:

``` python

#!mblist -Idatalistmergedshifted_0140-to-0144_ppppp.mb-1 -OXYZ > ODEMAR_Abyss_0140-0144_Irinovskoe_merged_centralbeam.xyz

```

%% Cell type:markdown id: tags:

*Enter narrative on the export here.

------

%% Cell type:code id: tags:

``` python

!mblist -Idatalistmergedshifted_0140-to-0144_ppppp.mb-1 -D3 -B2013/11/25/10/0/0 -E2013/11/27/10/0/0 > ODEMAR_Abyss_0140-0144_Irinovskoe_merged_0140.xyz

!mblist -Idatalistmergedshifted_0140-to-0144_ppppp.mb-1 -D3 -B2013/11/27/10/0/0 -E2013/11/30/10/0/0 > ODEMAR_Abyss_0140-0144_Irinovskoe_merged_0141.xyz

!mblist -Idatalistmergedshifted_0140-to-0144_ppppp.mb-1 -D3 -B2013/11/30/10/0/0 -E2013/12/02/10/0/0 > ODEMAR_Abyss_0140-0144_Irinovskoe_merged_0142.xyz

!mblist -Idatalistmergedshifted_0140-to-0144_ppppp.mb-1 -D3 -B2013/12/02/10/0/0 -E2013/12/04/10/0/0 > ODEMAR_Abyss_0140-0144_Irinovskoe_merged_0143.xyz

!mblist -Idatalistmergedshifted_0140-to-0144_ppppp.mb-1 -D3 -B2013/12/04/10/0/0 -E2013/12/05/10/0/0 > ODEMAR_Abyss_0140-0144_Irinovskoe_merged_0144.xyz

```

%% Cell type:code id: tags:

``` python

```

%% Cell type:code id: tags:

``` python

```