Commit f6b4bb83 authored by Meike Klischies's avatar Meike Klischies

added a description in the export section

parent 6197c299
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"AUV Abyss workflow for merging of multi-surveys\n",
"------\n",
"\n",
"M. Klischies, J. Mohrmann, I.A. Yeo, A. Steinführer, M. Rothenbeck\n",
"\n",
"\n",
"Workflow Version 2.0\n",
"\n",
"Aim:\n",
"\n",
"\n",
"Merge MBES data of a multi-survey mapping campaign to produce a consistent data set and a large-coverage map.\n",
"\n",
"-----\n",
"\n",
"**Workflow overview**\n",
"\n",
"Depending on the post-processing treatment of the individual surveys, the merging workflow branches into two different approaches.\n",
"\n",
"**A) Applies to renavigated and shifted data.** *Preferred option.*\n",
"- Prepare merge project folder\n",
"\n",
"ppp.mb88 and datalist_ppp.mb-1\n",
"- *mbnavadjust*\n",
" - Set up **merge project**\n",
" - Add data of two missions\n",
" - Fix navigation of each individual mission\n",
" - **Navigation adjustment in overlapping areas**\n",
"- *mbprocess* (_ppp.mb-1 datalists + ppp.mb88.na0 = pppp.mb88 files)\n",
"\n",
"pppp.mb88 -> datalist_pppp.mb-1\n",
"- Grid data to check quality\n",
"\n",
"ppp.mb88 and datalist_ppp.mb-1\n",
"- *mbnavadjust* (**Reopen** merge project)\n",
" - Add another mission\n",
" - Fix its navigation\n",
" - Renavigate overlapping areas with the other dives\n",
"- Repeat running *mbprocess* (_ppp.mb-1 datalists + ppp.mb88.na0 = pppp.mb88 files) and gridding data (pppp.mb88 -> datalist_pppp.mb-1) to check quality\n",
"- Reopen merge project and add another dives. Consecutively continue until all surveys are merged.\n",
"- *mbprocess* (_ppp.mb-1 datalists + ppp.mb88.na0 = pppp.mb88 files)\n",
"\n",
"pppp.mb88 -> datalist_pppp.mb-1\n",
"- Grid data to check quality\n",
"- Export of navigation adjusted, shifted and merged data\n",
"\n",
"**B) Applies to renavigated, but unshifted data.**\n",
"\n",
"pp.mb88 and datalist_pp.mb-1\n",
"- *mbnavadjustmerge* (combines existing navadjust projects and sets up an overall **merge project**)\n",
"- *mbnavadjust* (choose the created merge project to **adjust navigation in overlapping areas** and merge the surveys)\n",
"- *mbprocess* (_pp.mb-1 datalists + pp.mb88.na0 = ppp.mb88 files)\n",
"\n",
"ppp.mb88 -> datalist_ppp.mb-1\n",
"- Grid data to check quality\n",
"- Export navigation adjusted, merged data\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"## Target area\n",
"*(enter mapped area)*\n",
"\n",
"\n",
"Missions to be merged: *(list according surveys below)*\n",
"- 0XXX1\n",
"- 0XXX2 ...\n",
"\n",
"Enter short narrative on cruise/purpose/dive/circumstances/obstacles here.\n",
"\n",
"\n",
"Absolute navigation from: *(list georeferenced features with dive numbers below)*\n",
"- ... \n",
"- *(e.g. IODP## re-entry cone visible in dive 0XXX and 0XXX; fault scarp in dive 0XXX and 0XXX from ROV sampling)*\n",
"\n",
"------\n",
"\n",
"## Define basic parameters\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true,
"scrolled": true
},
"outputs": [],
"source": [
"rootpath=\"/enter/path/here\"\n",
"%cd $rootpath\n",
"\n",
"# define name of navadjust project (make sure this alings with what you enter in navadjust later)\n",
"navadjustProjectName = \"targetarea\"\n",
"\n",
"# datalists\n",
"datalist_navadjust = navadjustProjectName + \".mb-1\""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Prepare merge project\n",
"\n",
"Create a merge project folder containing the following:\n",
"\n",
"A)\n",
"- this jupyter notebook\n",
"- a copy of the dive files to be merged (_ppp.mb88 files)\n",
"- a copy of the corresponding datalists (_ppp.mb-1 datalists)\n",
"\n",
"B)\n",
"- this jupyter notebook\n",
"- a copy of the dive files to be merged (_pp.mb88 files)\n",
"- a copy of the corresponding datalists (_pp.mb-1 datalists)\n",
"- a copy of the existing, corresponding navadjust project files\n",
" - Abyss0XXX_navadj.mb-1\n",
" - Abyss0XXX_navadj.dir folder\n",
" - Abyss0XXX_navadj.nvh\n",
" - Abyss0XXX_navadj_dx.txt\n",
" - Abyss0XXX_navadj_dy.txt\n",
" - *.rte files\n",
"\n",
"\n",
"\n",
"-----\n",
"\n",
"## Merge project\n",
"\n",
"A) Data of each mission is shifted in respect to the above mentioned absolute navigation values, and/or referenced ship bathymetry (note the accuracies of available absolute navigation fixes!). So, each survey should be in the approximate right location. Disadvantage: set ties of previous navadjust-projects are lost. Therefore: **Prior to adjust overlapping areas, set ties within each survey to keep the previously adjusted navigation!**\n",
"\n",
"B) Use the tool *mbnavadjustmerge* to merge existing, not shifted navadjust-projects. Advantage: set ties of previous projects are translated. Disadvantage: only relative navigation constrains. **Limited to maximum offset of 400m between merged surveys!**\n",
"\n",
"-----"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Merge project - approach A)\n",
"\n",
"#### Mission 0XXX1 + 0XXX2\n",
"\n",
"*Enter short narrative of dive areas, overlap, and available navigation references.*\n",
"\n",
"\n",
"**Set up a new mbnavadjust project**\n",
"\n",
"- File > New > Name navadjust project: navadjustProjectName\n",
"- File > Import Swath Data > Choose datalistdive0XXX1\n",
"- File > Import Swath Data > Choose datalistdive0XXX2\n",
"\n",
"1) Set ties to keep internal navigation of each survey:\n",
"- View > Show Survey-vs-Survey Blocks > Choose Block survey 0 vs. survey 0 (subsequently 1 vs 1)\n",
"- View > Show Only Selected Survey-vs-Survey Block\n",
"- View > Show True Crossings > navadjust in NavErr window. Dismiss.\n",
"- View > Show Crossings with 50% overlap > navadjust in NavErr window. Dismiss.\n",
"- Action > Invert\n",
"\n",
"2) Merge overlapping areas:\n",
"- View > Show Survey-vs-Survey Blocks > Choose Block with survey0 vs survey1\n",
"- View > Show Only Selected Survey-vs-Survey Block\n",
"- View > Show True Crossings > navadjust in NavErr window. **Set ties carefully!** Dismiss. Repeat itinerary on more detailled levels, if neccessary.\n",
"- Action > Invert\n",
"- Action > Apply Adjusted Navigation.\n",
"- File > Quit.\n",
"\n",
"3) Check Quality, adjust, continue with adding a next mission.\n",
"\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"print(navadjustProjectName)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true,
"scrolled": false
},
"outputs": [],
"source": [
"!mbnavadjust"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true,
"scrolled": false
},
"outputs": [],
"source": [
"# process data using datalist automatically created by mbnavadjust\n",
"!mbprocess -IMerge_Irinovskoie.mb-1 -V"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true,
"scrolled": true
},
"outputs": [],
"source": [
"# generate new datalist of processed files\n",
"!ls -1 *pppp.mb88 > datalistmerged_0142-0143_pppp.mb-1"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true,
"scrolled": false
},
"outputs": [],
"source": [
"# Plot navigation \n",
"!mbm_plot -Idatalistmerged_0142-0143_pppp.mb-1 -N -OAbyss_0142-0143_merge-nav\n",
"!./Abyss_0142-0143_merge-nav.cmd"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true,
"scrolled": false
},
"outputs": [],
"source": [
"# Creating arcAscii grid of merged surveys\n",
"!mbgrid -Idatalistmerged_0142-0143_pppp.mb-1 -A2 -G4 -E3/0.0! -OAbyss_0142-0143_merge_3m"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Quality check of the adjusted navigation**\n",
"\n",
" - Navigation track is plausible and merged grid appears good: Continue with adding the next mission.\n",
" - Merged areas appear chaotic/erroneous: Carefully set some more ties.\n",
" - Renavigation causes a missfit in the survey, for which absolute navigation constrains are available: Set navigation of this survey as **good** and for the other ones as **poor**. Invert and apply navigation.\n",
" \n",
"**Intermediate result**\n",
"\n",
"*Enter narrative comment about navigation adjustment progress here.*\n",
"\n",
"\n",
"#### Mission 0XXX3\n",
"\n",
"*Enter short narrative of dive areas, overlap, and available navigation references.*\n",
"\n",
"**Adding data to existing mbnavadjust-project**\n",
"\n",
"- Re-open mbnavajust project\n",
"- File > Import Swath Data > Choose datalistdive0XXX3\n",
"\n",
"1) Set ties to keep internal navigation of each survey:\n",
"- View > Show Survey-vs-Survey Blocks > Choose Block survey 2 vs. survey 2\n",
"- View > Show Only Selected Survey-vs-Survey Block\n",
"- View > Show True Crossings > navadjust in NavErr window. Dismiss.\n",
"- View > Show Crossings with 50% overlap > navadjust in NavErr window. Dismiss.\n",
"- Action > Invert\n",
"\n",
"2) Merge overlapping areas:\n",
"- View > Show Survey-vs-Survey Blocks > Choose Block with survey0 vs survey 2.\n",
"- View > Show Only Selected Survey-vs-Survey Block.\n",
"- View > Show True Crossings > navadjust in NavErr window. **Set ties carefully!** Dismiss.\n",
"- Repeat itinerary on survey1 vs survey2 block.\n",
"- Action > Invert\n",
"- Action > Apply Adjusted Navigation.\n",
"- File > Quit.\n",
"\n",
"3) Check Quality, adjust, continue with adding a next mission."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true,
"scrolled": false
},
"outputs": [],
"source": [
"!mbnavadjust"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true,
"scrolled": false
},
"outputs": [],
"source": [
"# process data using datalist automatically created by mbnavadjust\n",
"!mbprocess -IMerge_Irinovskoie.mb-1 -V\n",
"\n",
"# new datalist for extended merging project\n",
"!ls -1 *pppp.mb88 > datalistmerged_0142-0143-0144_pppp.mb-1"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true,
"scrolled": false
},
"outputs": [],
"source": [
"# Plot navigation \n",
"!mbm_plot -Idatalistmerged_0142-0143-0144_pppp.mb-1 -N -OAbyss0242-to-0245_Cluster03_merge-nav\n",
"!./Abyss_0142-0143-0144_merge-nav.cmd\n",
"\n",
"# Creating arcAscii grid of merged surveys\n",
"!mbgrid -Idatalistmerged_0142-0143-0144_pppp.mb-1 -A2 -G4 -E3/0.0! -OAbyss_0142-0143-0144_merge_3m"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Quality check of the adjusted navigation**\n",
"\n",
" - Navigation track is plausible and merged grid appears good: Continue with adding the next dive.\n",
" - Merged areas appear chaotic/erroneous: Carefully set some more ties.\n",
" - Renavigation causes a missfit in the survey, for which absolute navigation constrains are available: Set navigation of this survey as **good** and for the other ones as **poor**. Invert and apply navigation.\n",
" \n",
"**Intermediate result**\n",
"\n",
"*Enter narrative comment about navigation adjustment progress here.*\n",
"\n",
"\n",
"#### Mission 0XXX4\n",
"\n",
"*Enter short narrative of dive areas, overlap, and available navigation references.*\n",
"\n",
"**Adding data to existing mbnavadjust-project**\n",
"\n",
"Continue as described above until all dives are added and merged."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"!mbnavadjust"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"# process data using datalist automatically created by mbnavadjust\n",
"!mbprocess -IMerge_Irinovskoie.mb-1 -V\n",
"\n",
"# new datalist for extended merging project\n",
"!ls -1 *pppp.mb88 >> datalistmerged_0141-0142-0143-0144_pppp.mb-1"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"# Plot navigation \n",
"!mbm_plot -Idatalistmerged_0141-0142-0143-0144_pppp.mb-1 -N -OAbyss_0141-0142-0143-0144_merge-nav\n",
"!./Abyss_0141-0142-0143-0144_merge-nav.cmd\n",
"\n",
"# Creating arcAscii grid of merged surveys\n",
"!mbgrid -Idatalistmerged_0141-0142-0143-0144_pppp.mb-1 -A2 -G4 -E3/0.0! -OAbyss_0141-0142-0143-0144_merge_3m"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Quality check of the adjusted navigation**\n",
"\n",
" - Navigation track is plausible and merged grid appears good: Continue with adding the next dive.\n",
" - Merged areas appear chaotic/erroneous: Carefully set some more ties.\n",
" - Renavigation causes a missfit in the survey, for which absolute navigation constrains are available (Check in GIS project): Set navigation of this survey as **good** and for the other ones as **poor**, then invert and apply navigation (in mbnavadjust).\n",
" \n",
"**Intermediate result**\n",
"\n",
"*Enter narrative comment about navigation adjustment progress here.*\n",
"\n",
"\n",
"#### Dive 0XXX4\n",
"\n",
"*Enter short narrative of dive areas, overlap, and available navigation references.*"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"!mbnavadjust"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true,
"scrolled": true
},
"outputs": [],
"source": [
"# process data using datalist automatically created by mbnavadjust\n",
"!mbprocess -IMerge_Irinovskoie.mb-1 -V\n",
"\n",
"# new datalist for extended merging project\n",
"!ls -1 *pppp.mb88 >> datalistmerged_0140-to-0144_pppp.mb-1"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true,
"scrolled": true
},
"outputs": [],
"source": [
"# Plot navigation \n",
"!mbm_plot -Idatalistmerged_0140-to-0144_pppp.mb-1 -N -OAbyss_0140-to-0144_Cluster03_merge-nav\n",
"!./Abyss_0140-to-0144_merge-nav.cmd\n",
"\n",
"# Creating arcAscii grid of merged surveys\n",
"!mbgrid -Idatalistmerged_0140-to-0144_pppp.mb-1 -A2 -G4 -E3/0.0! -OAbyss_0140-to-0144_merge_3m"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Merge project - approach B)\n",
"\n",
"#### Mission 0XXX1 + 0XXX2\n",
"\n",
"*Enter short narrative of dive areas, overlap, and available navigation references.*\n",
"\n",
"\n",
"**Setup a mbnavadjustmerge project**\n",
"1) Merge existing navadjust projects\n",
"- run mbnavadjustmerge command\n",
"\n",
"2) Merge overlapping areas\n",
"- File > Open > Choose navadjustmerge project\n",
"- View > Show Survey-vs-Survey Blocks > Choose Block with survey0 vs survey1\n",
"- View > Show Only Selected Survey-vs-Survey Block\n",
"- View > Show True Crossings > navadjust in NavErr window. **Set ties carefully!** Dismiss. Repeat itinerary on more detailled levels, if neccessary.\n",
"- Action > Invert\n",
"- Action > Apply Adjusted Navigation.\n",
"- File > Quit.\n",
"\n",
"3) Check Quality, adjust, continue with adding a next mission.\n",
"\n",
"**Adding data to existing mbnavadjust-project**\n",
"\n",
" ***CAUTION:***\n",
" *This workflow has NOT been tested for merging of more than two surveys!*\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true,
"scrolled": false
},
"outputs": [],
"source": [
"!mbnavadjustmerge --input=Abyss0XXX1-navadj1.nvh --input=Abyss0XXX2-navadj1.nvh --output=Abyss0XXX12_navadj1_merge.nvh"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"# process data using datalist automatically created by mbnavadjust\n",
"!mbprocess -IAbyss0XXX12_navadj1_merge.mb-1 -V\n",
"\n",
"# new datalist for extended merging project\n",
"!ls -1 *ppp.mb88 >> datalistmerged_Abyss0XXX12_ppp.mb-1"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true,
"scrolled": true
},
"outputs": [],
"source": [
"# Plot navigation \n",
"!mbm_plot -Idatalistmerged_Abyss0XXX12_ppp.mb-1 -N -OAbyss0XXX12_merge-nav\n",
"!./Abyss_0140-to-0144_merge-nav.cmd\n",
"\n",
"# Creating arcAscii grid of merged surveys\n",
"!mbgrid -Idatalistmerged_Abyss0XXX12_ppp.mb-1 -A2 -G4 -E3/0.0! -OAbyss0XXX12_merge_3m"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"**Result**\n",
"\n",
"*Enter narrative comment about navigation adjustment progress here.*\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"--------\n",
"\n",
"### Result of the merge project\n",
"\n",
"Appraoch A/B was applied, because ...\n",
"\n",
"The merged data is ...\n",
"\n",
"*Enter narrative on the quality of the navigation adjustment here.*\n",
"\n",
"-----\n",
"\n",
"\n",
"## Shift merged dataset\n",
"\n",
"\n",
"*Operator: (enter name here)*\n",
"\n",
"20yy-mm-dd\n",
"\n",
"\n",
"\n",
"\n",
"Horizontal shift is applied to the entire data set to match other, georeferenced locations or data.\n",
"\n",
"\n",
"**Determine the horizontal shift in a GIS project**\n",
"\n",
"Shift grid (datalistpppp) to match georeferenced data; enter distance and bearing below."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"# enter bearing and distance of shift according to shift of ascii-grid in GIS\n",
"dist=25 # distance in m\n",
"bear=10 # bearing in degree\n",
"\n",
"import numpy as np\n",
"\n",
"x_shift=np.sin(np.radians(bear))*dist\n",
"y_shift=np.cos(np.radians(bear))*dist\n",
"\n",
"print (x_shift)\n",
"print (y_shift)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"# set offset in parameter file of merged data\n",
"!mbset -Idatalistmerged_0140-to-0144_pppp.mb-1 -PNAVSHIFTX:$x_shift -PNAVSHIFTY:$y_shift \n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"# apply shift to merged data\n",
"!mbprocess -Idatalistmerged_0140-to-0144_pppp.mb-1 -V"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"# Create new datalist --> neccessary for merging project\n",
"!cd $rootpath\n",
"!ls -1 *ppppp.mb88 > datalistmergedshifted_0140-to-0144_ppppp.mb-1\n",
"\n",
"# Plot navigation track after shift\n",
"!mbm_plot -Idatalistmergedshifted_0140-to-0144_ppppp.mb-1 -N -OAbyss0140-0144_Irinovskoie_mergedshift\n",
"!./Abyss0140-0144_Irinovskoie_mergedshift.cmd\n",
"\n",
"# Creating Ascii grid of adjusted and shifted data to check shift\n",
"!mbgrid -Idatalistmergedshifted_0140-to-0144_ppppp.mb-1 -A2 -G4 -E3/0.0! -OAbyss0140-0144_Irinovskoie_mergedshift_3m"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Final result\n",
"\n",
"*Enter narrative on the final result of the navigation adjustment here.*\n",
"\n",
"\n",
"\n",
"-------\n",
"\n",
"## Converting .mb88 Files into .xyz-data\n",
"\n",
"\n",
"To clean the navigation adjusted, shifted data, the sounding information must be exported to a file format readable by common MBES processing software, such as Qimera, or CARIS HIPS."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true,
"scrolled": false
},
"outputs": [],
"source": [
"!mblist -Idatalistmergedshifted_0140-to-0144_ppppp.mb-1 -D3 > ODEMAR_Abyss_0140-0144_Irinovskoe_merged.xyz"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true,
"scrolled": false
},
"outputs": [],
"source": [
"#!mblist -Idatalistmergedshifted_0140-to-0144_ppppp.mb-1 -OXYZ > ODEMAR_Abyss_0140-0144_Irinovskoe_merged_centralbeam.xyz"
]
},
{
"cell_type": "markdown",
"metadata": {