In our last blog post, we established a remote connection to a Linux device running PowerShell Core. We did this using the PowerShell Core cmdlet New-PSSession and using SSH as the underlying protocol for the connection. The next big question is how can I utilize this remote connection to issue Cohesity CLI commands to the nodes in the cluster?
Simply, you can run Linux executables through this tunnel. So, you can actually run the ssh command in the remote connection tunnel! For the proper shutdown of a Cohesity system, it’s recommended that you stop the cluster services. If you’ve ever performed this operation before, you know this involves using our CLI interface on our nodes called iris_cli. We can issue the command to just a single node of the cluster and we can monitor the shutdown of the cluster services using the CLI interface.
In the case of the Mobile EBC, we had two clusters. In a PowerShell script that was stored on the remote tunnel device, I connected to a single node for each cluster and performed the cluster stop command using iris_cli.
ssh cohesity@coh01-node01.ebc.cohesity.mobi “iris_cli -username admin -password <password> -domain LOCAL cluster stop”
ssh cohesity@coh02-node01.ebc.cohesity.mobi “iris_cli -username admin -password <password> -domain LOCAL cluster stop”
With these two commands, we’ve connected to a member of each cluster and logged into our system to issue a cluster stop command. In my script, I did a Start-Sleep cmdlet call for 120 seconds (two minutes) to ensure the cluster services went down without a hitch. After this, I created a quick and dirty array to store the node numbers, which made it easy to loop through each node and issue the next command. I did this by performing the following:
$node = @()
$node += 1..8
The first code line creates an empty array. The second line says to add all values, in a series from 1 through 8, meaning our array should contain values for each number from one to eight.
Now that we have the node numbers handy, we can call each node in the cluster and perform a remote shutdown command. Keep in mind, this command issues a Power Off and Shutdown command to the node, so tread carefully with it:
$nodes | ForEach-Object {
ssh cohesity@coh01-node0$($_).ebc.cohesity.mobi “nohup /home/cohesity/bin/product_helper –op=SHUTDOWN </dev/null >/dev/null 2>&1 &”
}
So, there’s quite a bit going on here. If you are familiar with any Linux bash scripting, the end parts of the command line are used to essentially tell SSH to deliver the payload then disconnect immediately. The idea here was to issue the shutdown command and not get stuck in a wait state waiting for the command to time out because you were now stuck monitoring the full shutdown of the system.
At this point in the truck, I would start to see power lights going out on the Cohesity nodes, which was always a good thing to see when you may have to perform an emergency shutdown and have a limited amount of time on your battery backup units.
The last thing that would run in the truck after this was to power off the final necessary virtual machines (things like Active Directory controllers, DNS controllers, VMware NSX controllers, and Edge devices). Once this last stage completed, the process would have shut down all 24 physical Dell PowerEdge devices (running VMware ESXi and Microsoft Hyper-V) and 16 Cohesity nodes. A manual flip of a couple of power switches on the battery backup units and everything was powered down in the back of the truck and ready for the final physical security checks before being locked up and the truck moving onto its next destination!
Thank you for reading through this blog series on the Cohesity Mobile EBC shutdown procedure, and I encourage you to head over to the Cohesity Developer website to learn more about our integrations with other automation frameworks and our SDKs so that you can begin automating your own Cohesity environment.
