Docker Engine Enterprise and Mirantis Container Runtime for Business Central Developers

by Steve Endow

NOTE:  This post relates to Docker Engine Enterprise and Mirantis Container Runtime on Windows Server.  I don't believe this relates to or affects Docker Desktop.  But let me know if I missed anything regarding Docker Desktop, and let me know if I have any details wrong regarding Mirantis Container Runtime and its use with BcContainerHelper and Business Central Docker containers.

On September 27, 2021, Tobias Fenster shared a Microsoft announcement about the future of Docker Engine Enterprise, sometimes referred to as "Docker EE".  

https://twitter.com/tobiasfenster/status/1442533006812631047?s=20

This Microsoft announcement comes almost 2 years after Docker was acquired by a company named Mirantis, in November 2019.

The Microsoft announcement indicates that technical support for the Mirantis Container Runtime will be transitioned to Mirantis.

Mirantis also published a post explaining the transition, and discussing options for receiving support for Mirantis Container Runtime (aka "MCR"), and free licenses for up to 9 "nodes" for Windows Server users.

So what does this mean for Business Central developers who use Windows Server?

When I first read the Mirantis blog post, I assumed I would need to contact Mirantis to take advantage of their "special offer" to receive a free license for MCR for my Business Central development on Windows Server.

This page on the Mirantis web site explains the "beta pricing" offer is available until December 31, 2021.

I thought that I would request licenses for a few nodes, which would cost me $0.  But, that would NOT include any support.  

I'm okay with the $0 price tag, and if I did need support for some reason, I would be open to paying a reasonable price for support.  I would even be open to paying $50 per node to receive limited support, but based on the information on the web page shown above, it is not included by default with the 1-9 or 10-50 note packages by default.  But it may be possible to purchase support separately for 50 nodes or less.

I'm assuming that after December 31, 2021, Mirantis will come up with new pricing, and MCR will not be free for Windows users.

What about "Moby"?

But today I re-read the Mirantis blog post and noticed this.

Wait, what?

I'm a Business Central "developer".  I am not using my containers in "production".

Does that mean that I can use the "default Moby engine"?  Whatever that means?

I posted my initial thoughts about this to the hive mind called Twitter:

https://twitter.com/steveendow/status/1445494238955720713

I tagged our resident Business Central Docker expert, Tobias Fenster to see if I was missing anything.

I'm glad I tagged him, because he shared some important information about how Docker works and additional considerations about the Mirantis announcement that I believe are pretty important to Business Central developers.

Tobias replied, and mentioned that the free Moby engine has "No client component".

I didn't know what this meant, so he generously explained further.

So while we could run BC containers using Moby, technically we would not have a docker.exe "client" to manage or create the containers on Windows Server.

It seems we will need MCR

You can read the rest of the Twitter thread if you would like, but after a few more questions from me, and answers from Tobias, these are the points I think are important for BC developers who use Docker Engine Enterprise on Windows Server.

1. If you do NOT get a license for the Mirantis Container Runtime (MCR), and choose to use the "Moby engine", that is ONLY the "server" (daemon) portion of the solution.  Moby will allow you to host containers, but technically, that does not include the docker.exe "client" or any tools or utilities to help you manage your containers.  You could try to find other Docker tools, but Tobias suspects that limited interest in Docker on Windows means limited options for Docker tools on Windows.

2. As a Business Central developer, I regularly use the docker.exe "client" command in PowerShell to manage containers and images.  So I believe I will still need that, and therefore will need MCR.

3. The BcContainerHelper PowerShell module that BC developers use to create Business Central containers relies on the docker.exe "client" tool to create and maintain and interact with containers, so I assume that the full MCR solution will be required in order to continue using BcContainerHelper.  (there may be other ways to download a docker.exe client or equivalent, but I have no yet researched that)

Given this, I will want to request the MCR licenses from Mirantis so that I can have access to the full MCR product suite on my Windows Servers and continue to reliably use BcContainerHelper.

If you are aware of other solutions that provide a Docker client and related tools on Windows Server and work well with BcContainerHelper, please let me know.

Steve Endow is a Microsoft MVP in Los Angeles.  He works with Dynamics 365 Business Central, Microsoft Power Automate, Power Apps, Azure, and .NET.

You can also find him on Twitter and YouTube

Using $metadata and $expand with Business Central Web APIs

By Steve Endow

In June 2021, I discussed my circuitous journey to learn about and understand the $metadata and $expand OData "query options" and how to use them with the Business Central web API.

BC Web API OData Entity Model

Here is that video:

https://www.youtube.com/watch?v=ONYgG28r4vg

Today I had to research the $expand option for a BC API, but completely forgot how to use $metadata and find the related record "multiplicities" or "NavigationProperties" available for BC records exposed by APIs.

Since I had forgotten everything I discussed in my video from June, I figured it was time to write down a few notes in a blog post so that I can come back to this post in the future to refresh my memory.

The $metadata Option

When working with the Business Central web API, you can add $metadata to the end of the base API URL to get information about all of the available endpoints as well as related data elements that can be accessed using the $expand option.

For example, when working with my BC Docker Container named "dev18", I would use this URL:

http://dev18-default:7048/BC/api/v2.0/$metadata

When working with a BC SaaS environment:  (substitute your tenant and environment)

https://api.businesscentral.dynamics.com/v2.0/mycompany.com/PRODUCTION/api/v2.0/$metadata

You can read all the gory details in the OData documentation here:

http://docs.oasis-open.org/odata/odata/v4.0/os/part2-url-conventions/odata-v4.0-os-part2-url-conventions.html#_Toc372793772

If you access the above Business Central API URL with the $metadata option, the request should return a massive pile of XML that looks something like this:

BC Web API OData Entity Model

Okay, so what are you supposed to do with those 4,000 lines of XML?

First, find the BC API object that you are working with.  For example, let's suppose we want to know which additional data elements can be expanded on the Customer object.

If you search in this XML for "customer", you'll find lots of entries.  Look for the line that says EntityType Name="customer".

This shows the customer object and the properties (fields) that are returned by the API.

If you scroll down past the Properties, you will start to see "NavigationProperty" items.

These are the "child records" or "related records" that you can include with the Customer records using the $expand option.

The $expand Option

Now that we know the NavigationProperty objects that can be retrieved for a Customer, we can use the $expand option in our API request.

You can read all the gory OData details about $expand here:

http://docs.oasis-open.org/odata/odata/v4.0/os/part2-url-conventions/odata-v4.0-os-part2-url-conventions.html#_Toc372793860

For example, if I want to retrieve a specific customer, and I want to also retrieve the Contacts associated with the customer, my request would look like this:

Sample Docker container URL:

http://dev18-default:7048/BC/api/v2.0/companies(aaaa3def-4c1f-ec11-bb7c-000d3a2bffff)/customers(11113516-4d1f-ec11-bb7c-000d3a2b9999)?$expand=contactsInformation

Sample BC SaaS URL:

https://api.businesscentral.dynamics.com/v2.0/mycompany.com/PRODUCTION/api/v2.0/companies(aaa73282-83ff-ea11-aa61-0022481e3fff)/customers(1111895e-84ff-ea11-aa61-0022481e9999)?$expand=contactsInformation

Using this $expand option includes contactsInformation in the customer data:

If I repeat this process for BC items, I go back to my $metadata XML and search for EntityType Name="item".  I can then see the NavigationProperty values that are available for BC items.

In the case of items, there are only 5 navigation properties available for use with $expand.  Which is a bummer, because my customer would like to retrieve item quantities by location from the BC API.  Based on this metadata, it looks like quantities by location are not available, and it will likely require a custom API.

Steve Endow is a Microsoft MVP in Los Angeles.  He works with Dynamics 365 Business Central, Microsoft Power Automate, Power Apps, Azure, and .NET.

You can also find him on Twitter and YouTube

Automatically Create BC Docker Container - PowerShell Script version 3

 By Steve Endow

I created this PowerShell script to automatically re-create my Business Central Docker Containers every morning.

Version 3 of the script records BC Image version information to a log file and to the notification email.

(Thanks to Kishor Mistry for the improved method to get the BC image version!)

Email notification with full log

 #v3.0 - June 17, 2021  
 #  
 #Define the host, container, and image names  
 $server = $env:COMPUTERNAME  
 $containerName = "dev18"  
 $imageName = "i" + $containerName  
 $dnsIP = "192.168.25.21"  
   
 #Define date values  
 $simpleDate = Get-Date -Format "M/d/yy"  
 $dateTime = Get-Date -Format "M/d/yy HH:mm"  
 $fileDateTime = Get-Date -Format "yyyy-MM-dd HHmm"  
 $currentDate = Get-Date -Format "yyyy-MM-dd"  
 $currentTime = Get-Date -Format "hh:mm:ss"  
   
 #Specify file location for activity transcript log file  
 $transcriptFile = "D:\BCPowerShell\Logs\" + $fileDateTime + " " + $containerName + " Container Build Log.txt"  
   
 #File location for BC version tracking  
 $versionFile = "D:\BCPowerShell\Logs\" + $containerName + " BC Version Log.txt"  
   
 #Start recording transcript  
 Start-Transcript -Path $transcriptFile  
   
 #Record the start time  
 $StartTime = $(get-date)  
   
 #Define email configuration  
 $emailFrom = "precipioazure@gmail.com"  
 $emailTo = "steveendow@gmail.com"  
 $smtpServer = "smtp.gmail.com"  
 $port = "587"  
 #Email password file created using: Read-Host -AsSecureString | ConvertFrom-SecureString | Out-File -FilePath D:\BCPowerShell\Gmail.securestring  
 $passwordFile = "D:\BCPowerShell\Gmail_sendow.securestring"  
 $securePassword = ConvertTo-SecureString (Get-Content -Path $passwordFile)  
 $smtpCred = New-Object -TypeName PSCredential ($emailFrom, $securePassword)  
   
 $subject = $server + ": " + $simpleDate + " BC Build Log for Container: " + $containerName  
   
 $started = "Start time: " + $StartTime  
 $body = $started + "`n"  
   
 $errFlag = $false  
 $warnFlag = $false  
   
 #Output startup info  
 Write-Output $subject  
 Write-Output $started  
   
 Try {  
   
   #********************************************************************************************  
   #Get image version from current container  
   try {   
     $currentImageNum = (Get-BcContainerAppInfo -containerName $ContainerName | Where-Object -Property Publisher -EQ "Microsoft" | Where-Object -Property Name -EQ "Application").Version.ToString()   
   }  
   catch   
   {  
     $currentImageNum = 'No Container'  
   }  
   #********************************************************************************************  
   
   
   Remove-Module BcContainerHelper -ErrorAction SilentlyContinue  
   Import-Module BcContainerHelper -ErrorAction SilentlyContinue  
   
   #Remove old artifacts and images  
   Flush-ContainerHelperCache -cache bcartifacts -keepDays 7  
   
   #Remove existing container if it exists  
   Remove-BCContainer $containerName  
   
   #Remove old images without deleting base OS image  
   docker images --format "{{.Repository}}\t{{.Tag}}\t{{.ID}}" |    
    Select-String "businesscentral" -notMatch |   
    ConvertFrom-CSV -Delimiter "`t" -Header ("Repository","Tag","ID") |   
    Sort-Object Tag | % ID | % { docker rmi $_ }   
     
   #Call New-BcContainer - Use New-BCContainerWizard to generate the script/params for your environment  
   $password = 'P@ssw0rd'  
   $securePassword = ConvertTo-SecureString -String $password -AsPlainText -Force  
   $credential = New-Object pscredential 'admin', $securePassword  
   $auth = 'UserPassword'  
   $artifactUrl = Get-BcArtifactUrl -country us -select Latest -type Sandbox  
   New-BcContainer `  
     -accept_eula `  
     -containerName $containerName `  
     -credential $credential `  
     -auth $auth `  
     -artifactUrl $artifactUrl `  
     -dns $dnsIP `  
     -updateHosts `  
     -licenseFile "D:\BCPowerShell\BCLicense\BCv18.flf" `  
     -assignPremiumPlan `  
     -includeTestToolkit `  
     -includeAL `  
     -shortcuts None `  
     -imageName $imageName  
       
   
   Publish-BcContainerApp -appFile "D:\zDisks\BCCL\BCCL_1.15.0.125\installer\Extension\Hougaard.com_Business Central Command Line_1.15.0.125.app" -containerName dev18 -install -packageType Extension -sync -syncMode Add  
   
   }  
   Catch {  
     #If the script fails or has an error, output to error stream  
     Write-Error "Um, something didn't go well:"  
     Write-Error $_  
     #Capture error in email body   
     $body += "`nERROR:`n`n" + $_ + "`n`n"  
     $errFlag = $true  
   }  
   Finally {  
       
   
     #********************************************************************************************  
     #Get image version info from new container  
     try {   
       $newImageNum = (Get-BcContainerAppInfo -containerName $ContainerName | Where-Object -Property Publisher -EQ "Microsoft" | Where-Object -Property Name -EQ "Application").Version.ToString()   
     }  
     catch   
     {  
       $newImageNum = 'No Container'  
     }  
   
     #Create line to write to BC Version log file  
     $versionEntry = $currentDate.ToString() + "`t" + $currentTime.ToString() + "`t" + $currentImageNum + "`t" + $newImageNum  
   
     #Write entry to BC Version log file  
     Add-Content -Path $versionFile -Value $versionEntry  
   
     #Add BC Versions to email body  
     $body += "`n" + "Prior BC version: " + $currentImageNum  
     $body += "`n" + "New BC version: " + $newImageNum + "`n"  
     #********************************************************************************************  
   
   
     #PROTOTYPE - This is fairly fragile and needs improvement  
   
     #Search for the Container OS Build number and compare it to the Host OS build number  
     [string]$containerOSVersion = Select-String -Pattern "Container OS Version: " -SimpleMatch -Path $transcriptFile  
     $trimBefore = $containerOSVersion.IndexOf("Version: ")  
     #This is pretty crude parsing, so there is presumably a more elegant method  
     $containerOSBuild = $containerOSVersion.Substring($trimBefore+9, ($containerOSVersion.Length-($trimBefore+9))).Split(" (")[0].Split(".")[3]  
     $osVersion = (Get-ItemProperty -Path "HKLM:\SOFTWARE\Microsoft\Windows NT\CurrentVersion" -Name UBR).UBR  
     If ($containerOSBuild -ne $osVersion)  
     {  
       $warnFlag = $true  
       $buildWarning = "`nWARNING: Container OS Build " + $containerOSBuild + " does NOT match Host OS Build " + $osVersion  
       Write-Error $buildWarning  
       $body += "`n" + $buildWarning + "`n"  
     }  
     Else  
     {  
       $output = "Container OS Build " + $containerOSBuild + " matches Host OS Build " + $osVersion  
       Write-Output $output  
       $body += "`n" + $output + "`n"  
     }  
   
     ipconfig /flushdns;  
   
     #Record end time  
     $EndTime = $(get-date)  
     $output = "End time: " + $EndTime  
     $body += "`n" + $output   
     Write-Output $output   
   
     #Calculate elapsed time  
     $elapsedTime = $EndTime - $StartTime  
     $totalTime = "{0:HH:mm:ss}" -f ([datetime]$elapsedTime.Ticks)  
     $output = "Elapsed time: " + $totalTime  
     $body += "`n" + $output   
     Write-Output $output  
   
     $body += "`n`nFull log file is attached"   
   
     If ($errFlag)  
     {  
       $subject = "ERROR: " + $subject  
       $greeting = "Greetings human.`n`nIt appears there was an error building your Business Central Docker Container: " + $containerName + "`n`n"  
     }  
     elseif ($warnFlag)  
     {  
       $subject = "WARNING: " + $subject  
       $greeting = "Greetings human.`n`nI prepared your Business Central Docker Container: " + $containerName + ", but detected potential issues.`n`n"  
     }  
     else  
     {  
       $greeting = "Greetings human.`n`nI have prepared your Business Central Docker Container: " + $containerName + "`n`n"  
     }  
   
     $body = $greeting + $body  
   
   
     #Finish the transcript recording  
     #The Transcript must be stopped before trying to send the email,  
     #otherwise the log file will be locked, causing the email to fail  
     Stop-Transcript  
       
     #Send the email  
     Send-MailMessage -From $emailFrom -To $emailTo `  
       -Subject $subject `  
       -Body $body `  
       -Attachments $transcriptFile `  
       -SmtpServer $smtpServer `  
       -Credential $smtpCred `  
       -Port $port `  
       -UseSsl  
             
   }  
   
   

Steve Endow is a Microsoft MVP in Los Angeles.  He works with Dynamics 365 Business Central, Microsoft Power Automate, Power Apps, Azure, and .NET.

You can also find him on Twitter and YouTube

Restoring data from a broken Business Central Docker Container

 By Steve Endow 

(This blog post is inspired by a forum question. Credit to Arvydas for the question and sharing the solution.)

What if I made the mistake of storing some "worthwhile" data in a Business Central Docker container over the course of several months?  Perhaps it was test data or data imports that I didn't think I would ever need--but then 3 months later I realized that the accumulated data was valuable.

In theory, no big deal.  If you realize that you want to keep the data from a working container, Stefan Maroń has a blog post with instructions on converting that Container into an Image using "docker commit".  Just create an image and you can make new containers from that image.  You could also backup the SQL databases from the Docker Container to the local file system.  

But, what if the Business Central Docker Container with your valuable data stops working due to a Windows Update?  And you only realize you need that data after the container stops working.  If the Container can't be started at all, how can you extract the Business Central data?

This solution appears to allow me to copy the SQL mdf and ldf files from the broken container to my local file system, then copy the mdf and ldf files to a new working container to effectively "restore" them.

This may not work for all situations where a BC container has issues, but should work if the container is intact, but just won't start up due to a Docker or Windows issue or error.  

Let me know if there are any issues or caveats with using this method to restore the data from a container.

Software Algorithms to Solve Messy Data Problems

 By Steve Endow

In the software / consulting world, we're sometimes given data that is messy, and we, the software developers, consultants, or report writers, are tasked with making sense of that data and using it in a business process.  Sometimes we have to do "creative" and often unintuitive things to that data to make it usable.

For example, I was once asked to import employee time clock data into a payroll system.  Simple, right?  In the ideal world, an employee clocks in for work in the morning, and clocks out for work in the afternoon.  But in the real world, it's never this easy.

First, the customer's horrible timeclock system did not differentiate between a "clock in" or a "clock out".  I was simply given two fields:  Employee ID and Date Time.  It was up to me to figure out which entries were "clock in" and which entries were "clock out".

Second, sometimes the employees would take breaks or leave for lunch. They would therefore "clock out" when leaving, and "clock in" when returning to work. So an employee could have up to four records for a single shift.

Automated PowerShell Script v2.1 to Recreate My Business Central Docker Containers

 By Steve Endow

UPDATE:  I have posted version 3 of the script here

In July 2020 I posted an earlier version of this script.  This updated version has a few refinements, such as removing and importing BcContainerHelper to ensure the latest version is being used, adding Flush-ContainerHelperCache to remove old helper files, and removing old Docker images.

Automate, Automate, Automate

I setup a "Dev Container" task in Windows Task Scheduler to run this script every morning at 6:45am so that I always have a fresh BC container based on the latest BC artifacts.  I call my main Container "dev17", based on the current 17.x version of Business Central.  Do NOT name your container just "dev"--see my video here on why that will cause problems.

Changing the network location from Public to Private on Windows Server

By Steve Endow

This is just a post for me to store this information for Future Steve, as he will most certainly need it, just as Present Steve has needed it a dozen times over the last year.

Any time I replace my router, replace a piece of Ubiquiti network gear, or reconfigure my network, all of my Windows Server VMs decide they've been kidnapped and change their network location to Public network.

This blocks all traffic on the servers until I can login directly to each server and fix the problem.

Which requires an annoying process using gpedit or PowerShell, which I can never remember after 6 months.

So here is the post with the information to fix the problem:

https://serverfault.com/questions/639088/how-do-i-make-a-connection-private-on-windows-server-2012-r2

Using the GUI:

1. Hit Winkey + R to open Run prompt and type gpedit.msc
2. Navigate to: Computer Configuration/Windows Settings/Security Setting /Network List Manager Policies.
3. Choose your Network name in the right pane.
   Note: To show networks not currently connected, right-click Network List Manager Policies in the left pane and choose Show All Networks.

4. Go to Network Location tab and change the Location type from Public to Private.

Using PowerShell...

One line version:

Set-NetConnectionProfile -InterfaceAlias Ethernet -NetworkCategory Private

Multi-line version:

PS C:\>$Profile = Get-NetConnectionProfile -InterfaceAlias Ethernet1

PS C:\>$Profile.NetworkCategory = "Private"

PS C:\>Set-NetConnectionProfile -InputObject $Profile

Steve Endow is a Microsoft MVP in Los Angeles.  He works with Dynamics 365 Business Central, Microsoft Power Automate, Power Apps, Azure, and .NET.

You can also find him on Twitter and YouTube