Monday, February 26, 2018

Replacing a Veeam Agent for Windows host while preserving existing local or share backups

Replacing a Veeam Agent for Windows host while preserving existing local or share backups
// StarWind Blog

No ratings yet.


Imagine you have a server with data source volumes that are backed up to local (or a share) target backup volumes with Veeam Agent for Windows (VAW). You might or might not backup the OS as well. That server is old, has issues or has crashed beyond repair and needs to be replaced. You don't really care all that much about the server OS potentially but you do care about your data backup history! You don't want to lose all those restore points. Basically, we try to answer how do you replace the backup server when it's a local Veeam Agent for Windows 2.1 deployment.

How do you do this? Well, you're out of luck a bit unless you have been sending the backups to a VBR repository. With local (or share) backup targets you have a challenge when the host changes, as the unique identifier changes. But that challenge can be met successfully. We can access the localdb of VAW as the password is known and stored in the registry. So potentially you can swap over the unique identifier of the backups that associates a repository with the host. But in the case of local backups you need to recuperate the entire local database anyway. And as the host is "This server" and not the server name that might work without even needing to edit the database. At least that's what I learn from looking around a bit in the localdb.

replacing the backup server when a local Veeam Agent for Windows 2.1 deployment

The only place where host seems to truly identify is in names of backup files and in backup target file shares (host name or IP). So, the question is if we can complete a VAW SQL localdb transport successfully. As it turns out we can. That procedure is explained below. Now I'm not promoting this but I'm sharing is as it is part of my research to know how what to do when a host being backup with VAW to whatever repository (local, share, VBR) needs to be replaced.

WARNING: This is not a supported process by Veeam. It's just what I have come up with when faced with this challenge for this particular use case. No guarantees are given. The best advise I can give is to test this out in a virtual lab before using it for real.

As stated we are not super interested in the server OS actually as that can be easily replaced as long as you have some information about your existing VAW deployment at hand. If the server is dead that info might be the only info you want out of the OS backup. If you do have the OS backup in VAW backups that's easily dealt with in this scenario. Normally you're not interested in the backup history of the OS as you are replacing it with other hardware (or a new virtual machine).

The drive letters of the original LUNs.

The following information from the registry on the old server: in the following key HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Endpoint Backup\DbConfiguration we need the value for the SQL password string.

That's it. That's all you need. It would be smart to have this info somewhere in case of a total loss server but you could grab it out of an OS backup if you have one. Otherwise, you can just take a peek at the original server for that info. Which is what we'll do here as a demo.

Note: with VEB the windows LUN ID/disk order was important as well as the disk IDs in windows of those original LUNs. In VAW it seems less finicky about those, but it's good to have them just in case. I have tested this procedure with different LUNs and I had no issues even when moving around disks to other controllers, adding different disks etc.

Replacing a Veeam Agent for Windows Server while preserving existing local backups

We'll walk through this a lab exercise. On the original server, we have some data on the SOURCE volume that we backup to the TARGET volume on our lab server. We've configured a volume level backup.

replacing a Veeam Agent for Windows Server while preserving existing local backups - configured a volume level backup

We configure VAW 2.1 to backup volume S: (SOURCE) to volume T: (TARGET). We run the backup 5 times. This creates 1 full and 4 incremental files.

replacing a Veeam Agent for Windows Server while preserving existing local backups - configuretion VAW 2.1 to backup volum - running the backup 5 times

You can look at the backup files on the target and you'll see the metadata file (vbm), the full backup file (vbk) and the incremental backup files (vbi). Please note that the server host name is used in the file names!

replacing a Veeam Agent for Windows Server while preserving existing local backups - backup files

So how do we replace that server with a new one? Just install one and add a brand new VAW installation to it. So now we want to move the source volume and the target volume to a new server. Just like you might want to replace a server with a new one after it has failed or when it needs to be upgrade but the data source and backup target volume are retained. The nature of the storage will determine how that is done (SATA, SAS, iSCSI, FC).

We now shut down the VAW service on both the original and the new server.

replacing a Veeam Agent for Windows Server while preserving existing local backups - shutting down the VAW service

Copy the SQL password from the original server under HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Endpoint Backup\DbConfiguration => SqlPassword string. For example: cc0b7b4a-362b-4fdb-a60b-b6d7bf2744cd

replacing a Veeam Agent for Windows Server while preserving existing local backups - SQL password

Paste it into that value on the new server. That will allow us to open the database of the old server that we will copy to the new server. If the server crashed you can grab the registry hive file from a backup (C:\Windows\System32\config\SOFTWARE) copy it to a windows machine and with a registry file reader (regedit on a working machine will do) and grab the value of SQLPassword.

Note: do not worry about the SQLinstancePipeName value. While you need this to connect to the VeeamBackup database it changes when the VAW service is restarted anyway and has no impact on your ability to backup.

You can find the database files under C:\Windows\System32\config\systemprofile.You're after VeeamBackup.mdf and VeeamBackup_log.ldf. Only SYSTEM has access to those files by default. You do not have ownership or any rights to those localdb files.

Note: don't work on the copies under C:\ProgramData\Veeam\EndpointData, those seem to be a copy of the ones under C:\Windows\System32\config\systemprofile. If you rename the copies under C:\ProgramData\Veeam\EndpointData you'll see after a reboot or service restart, a new version gets copied over there.

replacing a Veeam Agent for Windows Server while preserving existing local backups - new version copies

You don't have permissions, not even as administrator, on the VeeamBackup files

You either have to grab ownership and grant yourself the permission's (and fix it back afterwards). Another option is to use psexec -s -i cmd/exe and run the move/copy commands under a system in that window. I just copy them out of there into a temporary location on the old server and then copy them to the new one. That can be a USB drive or thumb drive, a share, whatever works. If the original server died you need to grab them from the most recent backup.

replacing a Veeam Agent for Windows Server while preserving existing local backups - permissions on the VeeamBackup files

Leverage psexec to run commands as SYSTEM which helps copy files you normally don't have access to without having to alter any permissions or ownership.

The host name of the server doesn't really seem to matter for this operation (identified as "This Server", remember). Now because the files VAW creates are based on the host name we can (re)name the new server to match the old name. For a domain member server this means the old one will be kicked out of the domain / renamed and the computer AD object will be reset unless that new server is in another domain. For standalone servers this is less complex. While this keeps things consistent, it is actually not needed. The backups will work on a host with a different name, and the new backup files names will start reflecting the new host name as you can see in the picture below! Pretty resilient stuff.

replacing a Veeam Agent for Windows Server while preserving existing local backups - leveraging psexec to run commands as system

In the database, we'll see the backup job name changes to the new server name if we don't recuperate the original host name.

replacing a Veeam Agent for Windows Server while preserving existing local backups - the new host name

The new backup file name uses the new host name if we don't recuperate the original host name.

The host name, the backup job name and the file paths of the vbm, vbf, vbi files (screenshot above) are not used as unique identifiers. Only when a volume unique identifier (a guid, the DeviceID) changes you'll run into issues. You can research this in the VeeamBackup database.

replacing a Veeam Agent for Windows Server while preserving existing local backups - the backup job name and the file paths of the vbm, vbf, vbi files

Looking around in the VeeamBackup database is educational. You can use free portable tools for this so don't need to install anything on the server. Use the named pipe and the login credentials you can read in the registry.

Now if you want the backup job folder name to reflect the new server name the only way I have found to do that is recuperating the old name. There are just too many referrals in the database to the path and of the backup files stored for the tasks, jobs, restores etc. They would all need to be changed if you can identify them all. So, that's not really an option – I didn't drive it that far anyway – I achieved my goals, this is cosmetics. If it was a good idea I'm sure Veeam would have built it into the product like they did for picking up the name change. In any case, for replacing a failed server recuperating the old server name is a good way and you get 100% naming consistency.

Finally, you grant ownership back to SYSTEM if you so desire and take away the rights you gave yourself to end up in a permissions situation you had at the start. If you did everything you needed to do via psexec under SYSTEM everything is already as it should be security wise.

The source data volumes and the backup target volumes must be brought offline and unpresented from the old host. We then present both the source data volumes and the backup target volumes to the new server. We bring these volumes online and make sure they get the same drive letter. In our case Source S: and Target T:

replacing a Veeam Agent for Windows Server while preserving existing local backups - Virtual Mashine conection

Start the Veeam Agent for Microsoft Windows service on the new server.

starting Veeam Agent for Microsoft Windows service

When that is running open op the VAW control panel. You will see the backup history you had on your old server pop up on your new server.

starting Veeam Agent for Microsoft Windows service - VAW control panel

Now we'll run a backup to prove that this works. You'll notice it does run a full back up as the digests have to be calculated again.

starting Veeam Agent for Microsoft Windows service - runing a full back up

The backup finished at 12:59:44 AM.

starting Veeam Agent for Microsoft Windows service - backup finishing

The next test is to prove we can still restore the older backups we test this with the backup taken at 12:10.

starting Veeam Agent for Microsoft Windows service - restoring the older backups

As you can see we are ready to restore historical data on our new server. Do note that restores historical backups for the OS volume are no longer that useful that when you have moved to new hardware. These will expire over time and be gone.

That's it. You have moved the source and data volume used by Veeam Agent for Windows 2.1 on the old server to the new server. By copying the localdb files over from the old server to the new server and editing the registry you were able to maintain your previous backups and have those available for restores. The first backup you create is a full one as it has to calculate the digests. But that's the price you pay to keep your backup history.

WARNING: This is not a supported process by Veeam. It's just what I have come up with when faced with this challenge for this particular use case.

Please rate this


Read in my feedly

Sent from my iPhone

Don’t break your fingers with hundreds of clicks – automate Windows iSCSI connections

Don't break your fingers with hundreds of clicks – automate Windows iSCSI connections
// StarWind Blog

No ratings yet.


Hi everyone,

I believe any person reading this might have had one's hands on connecting iSCSI targets in Microsoft environments. First, you add target discovery portals with about a dozen of button clicks, then you proceed with connecting the targets with even more dozens of clicks. What if I tell you there is another and far simpler way to do so?

Today, I would like to spend some time on automation of iSCSI connections in Windows. As in most cases, this can be easily achieved using PowerShell, the great automation tool that Microsoft offers to all of us.

Automating Windows iSCSI connections

The initial step is making sure that your server has got it's Microsoft iSCSI service running at startup. Run the below command and make it do so:

Set-Service -Name msiscsi -StartupType Automatic

Further on, we need to discover a target portal – at least one, local or remote, from which we need to get the targets connected. If we discover the loopback interface, the command to run is:

New-IscsiTargetPortal -TargetPortalAddress ""

Pretty straightforward, isn't it? Yet, this can be tricky for some setups. The explanation to this is that when discovering the target portal using the above command, the default iSCSI initiator will be used. In case the system has got more than one iSCSI initiator, this can cause you some headache. So, we need to clearly indicate the initiator to be used when discovering the target portal. To see the list of the initiators available, run the following using the elevated command prompt:

iscsicli listinitiators

For this article, I will use the Microsoft iSCSI initiator, which is ROOT\ISCSIPRT\0000_0 in our example:

Listing iSCSI initiators using CMD or PowerShell

Thus, the amended command for discovering target portals will look like below:

New-IscsiTargetPortal -TargetPortalAddress "" -InitiatorInstanceName "ROOT\ISCSIPRT\0000_0"

In case you need to discover an IP address that is different from the loopback interface, let's say IP needs to be discovered from IP, the commands will undergo further changes. Particularly, I would advise adding the InitiatorPortalAddress parameter. This is done to avoid a mess when establishing the iSCSI connections. The indication of this parameter will ensure correct connection has been made in any case.

New-IscsiTargetPortal -TargetPortalAddress "" -InitiatorPortalAddress "" -InitiatorInstanceName "ROOT\ISCSIPRT\0000_0"

After we have discovered the target portals, our next step is connecting the iSCSI targets exposed by them. The simplest way to connect all iSCSI targets from the discovered portals is by shooting the following:

Get-IscsiTarget | Connect-IscsiTarget

This will connect the targets, but be careful – the connections made with this command will not be reestablished upon a reboot. To make sure they will get reconnected after any server maintenance, the –IsPersistent $true parameter should be used.

Get-IscsiTarget | Connect-IscsiTarget –IsPersistent $true

Note: If the targets are not connected, this will connect all of them. If any of them is already connected, an error will be presented by the PowerShell window for that target.

Our setup procedure has become much better now, yet not perfect. If you require multipathing for your targets, the Set-MSDSMGlobalDefaultLoadBalancePolicy cmdlet can be used, which accepts the -Policy parameter for setting the default MPIO policy.

Set-MSDSMGlobalDefaultLoadBalancePolicy -Policy RR

Here, RR stands for Round Robin. For other options, feel free to check this page from Microsoft.

Generally, by now we have covered the basic procedures for discovering iSCSI target portals and connecting the targets presented by them. And here comes a little bonus. As I am working for StarWind as a technical support engineer, from time to time I need to play around with some test setups. Connecting a bunch of iSCSI targets is a tedious thing, I must confess. When you have at least 3 targets (CSV1, CSV2, and Witness) – to be further added as Cluster Shared Volumes and the Quorum respectively – for proper storage load balancing in the Windows Failover Cluster, connecting them correctly does take time. I do not even mention cases when, in addition to the three afore-mentioned targets you have a bunch of other, like CSV3 to CSV(N). In addition to the above, I adore automation and try to do that wherever possible. This is why I have developed a little script that does the magic instead of me and connects targets for StarWind HA devices (replicated between nodes VSAN1 and VSAN2) in the Microsoft iSCSI initiator.

Just a couple of remarks concerning the script:

  • Node names are expected to be ending in either "1" or "2" (e.g. VSAN1 / VSAN2). If not, rename them or adjust the script
  • Network interface(s) intended for iSCSI connections should have "iscsi" as a part of the label
  • Targets shall be named CSV1, CSV2, and Witness and end like "node_number-csv_csvnumber" (e.g. in "vsan1-csv1"). In fact, the digit in the CSV name does not really matter, but the "CSV" part is introduced into the script through a variable, as well as "Witness", so keep this in mind. You can change the hardcoded values as needed.

Little disclaimer:
I do know that hardcoding is a bad thing for the majority of scenarios. Yet, this is an example script to be extended, improved or adjusted by any reader that might come across this blog post. So, I would like to keep some opportunities for you, guys, to fuel your ego by something like "Dude, I can improve this!". 😊

# StarWind iSCSI connections automation script    # Developed by Boris Yurchenko, StarWind    # some variables declarations    $initiator = "ROOT\ISCSIPRT\0000_0"    $loopbackip = ""    $node1iscsi1 = ""    $node1iscsi2 = ""    $node2iscsi1 = ""    $node2iscsi2 = ""    $csvpartialname = "csv"    $witnesspartialname = "witness"    # clear any previous output from the PowerShell window    cls    write-host "----------------------------    This script will add ISCSI discovery portals to Microsoft ISCSI initiator and connect available disks using the settings for the best performance.    ----------------------------"    # getting to know the node index    $computername = (Get-WmiObject -Class Win32_ComputerSystem -Property Name).Name    $hostLength = $computername.Length    $nodeNumber = $computername.Substring($hostlength - 1)    write-host "Node index detected: $nodeNumber" -ForegroundColor Green    # discovering the network interfaces intended for iSCSI connections    $iscsiinterfaces = Get-WmiObject Win32_NetworkAdapter | Select-Object -ExpandProperty NetConnectionID    $iscsiinterfaces = $iscsiinterfaces -like "iscsi*"    $iscsinumber = $iscsiinterfaces.Count    write-host "$iscsinumber iSCSI interfaces discovered"    # turning on autostart for the Microsoft iSCSI service    write-host "Autostarting iSCSI service"    Set-Service -Name msiscsi -StartupType Automatic    write-host "Adding iSCSI discovery portals"    # adding loopback interface and partner node interface(s) to the discovered portals    New-IscsiTargetPortal -TargetPortalAddress $loopbackip -InitiatorInstanceName $initiator | out-null    if ($nodeNumber -eq "1") {    New-IscsiTargetPortal -TargetPortalAddress $node2iscsi1 -InitiatorPortalAddress $node1iscsi1 -InitiatorInstanceName $initiator | out-null    if ($ISCSIinterfaces.Length -eq "2") {    New-IscsiTargetPortal -TargetPortalAddress $node2iscsi2 -InitiatorPortalAddress $node1iscsi2 -InitiatorInstanceName $initiator | out-null    }    } elseif ($nodeNumber -eq "2") {    New-IscsiTargetPortal -TargetPortalAddress $node1iscsi1 -InitiatorPortalAddress $node2iscsi1 -InitiatorInstanceName $initiator | out-null    if ($ISCSIinterfaces.Length -eq "2"){    New-IscsiTargetPortal -TargetPortalAddress $node1iscsi2 -InitiatorPortalAddress $node2iscsi2 -InitiatorInstanceName $initiator | out-null    }    }    write-host "Connecting iSCSI targets from discovered portals"    $loopback = read-host "Number of loopback sessions to add"    $partner = read-host "Number of partner sessions to add (for each interface, if more than one iSCSI interface is available)"    # magic happening further    $localISCSItargets = Get-IscsiTarget    $targetsnumber = $localISCSItargets.Count    write-host "We have $targetsnumber targets to be connected"    for ($i=0; $i -lt $targetsnumber; $i++) {    $activetarget = $localISCSItargets[$i].NodeAddress    if ($nodeNumber -eq "1"){    if ($activetarget -Like "*1-$witnesspartialname*") {    Connect-IscsiTarget -NodeAddress "$activetarget" -TargetPortalAddress $loopbackip -InitiatorInstanceName $initiator -IsMultipathEnabled $true -IsPersistent $true    }    if ($activetarget -Like "*1-$csvpartialname*"){    for ($j=0; $j -lt $loopback; $j++) {    Connect-IscsiTarget -NodeAddress "$activetarget" -TargetPortalAddress $loopbackip -InitiatorInstanceName $initiator -IsMultipathEnabled $true -IsPersistent $true    }    }    if ($activetarget -Like "*2-$csvpartialname*") {    if ($iscsinumber -eq "1") {    for ($l=0; $l -lt $partner; $l++) {    Connect-IscsiTarget -NodeAddress "$activetarget" -InitiatorPortalAddress $node1iscsi1 -TargetPortalAddress $node2iscsi1 -InitiatorInstanceName $initiator -IsMultipathEnabled $true -IsPersistent $true    }    } elseif ($iscsinumber -eq "2") {    for ($k=0; $k -lt $partner; $k++) {    Connect-IscsiTarget -NodeAddress "$activetarget" -InitiatorPortalAddress $node1iscsi1 -TargetPortalAddress $node2iscsi1 -InitiatorInstanceName $initiator -IsMultipathEnabled $true -IsPersistent $true    Connect-IscsiTarget -NodeAddress "$activetarget" -InitiatorPortalAddress $node1iscsi2 -TargetPortalAddress $node2iscsi2 -InitiatorInstanceName $initiator -IsMultipathEnabled $true -IsPersistent $true    }    }    }    } elseif ($nodeNumber -eq "2") {    if ($activetarget -Like "*2-$witnesspartialname*") {    Connect-IscsiTarget -NodeAddress "$activetarget" -TargetPortalAddress $loopbackip -InitiatorInstanceName $initiator -IsMultipathEnabled $true -IsPersistent $true    }    if ($activetarget -Like "*2-$csvpartialname*"){    for ($z=0; $z -lt $loopback; $z++) {    Connect-IscsiTarget -NodeAddress "$activetarget" -TargetPortalAddress $loopbackip -InitiatorInstanceName $initiator -IsMultipathEnabled $true -IsPersistent $true    }    }    if (($activetarget -Like "*1-$csvpartialname*")) {    if ($iscsinumber -eq "1") {    for ($x=0; $x -lt $partner; $x++) {    Connect-IscsiTarget -NodeAddress "$activetarget" -InitiatorPortalAddress $node2iscsi1 -TargetPortalAddress $node1iscsi1 -InitiatorInstanceName $initiator -IsMultipathEnabled $true -IsPersistent $true    }    } elseif ($iscsinumber -eq "2") {    for ($y=0; $y -lt $partner; $y++) {    Connect-IscsiTarget -NodeAddress "$activetarget" -InitiatorPortalAddress $node2iscsi1 -TargetPortalAddress $node1iscsi1 -InitiatorInstanceName $initiator -IsMultipathEnabled $true -IsPersistent $true    Connect-IscsiTarget -NodeAddress "$activetarget" -InitiatorPortalAddress $node2iscsi2 -TargetPortalAddress $node1iscsi2 -InitiatorInstanceName $initiator -IsMultipathEnabled $true -IsPersistent $true    }    }          }    }    }    # setting MPIO load balance policy    Set-MSDSMGlobalDefaultLoadBalancePolicy -Policy LQD    write-host "    Press any key to exit"    $HOST.UI.RawUI.ReadKey("NoEcho,IncludeKeyDown") | Out-Null    $HOST.UI.RawUI.Flushinputbuffer()


Another small remark goes here. If you decide to assign a separate MPIO load balance policy for each device, unfortunately, it seems to be impossible with PowerShell, as MPIO policy is applied to all of them at the same time when done through PowerShell. So, guys, if any of you knows how to define a separate MPIO load balance policy for certain devices without affecting other ones, feel free to share this with me and the community in the comments below this post. I will really appreciate it, as this thing has been annoying me for quite a period of time already.


If you have a single environment with only several iSCSI targets discovered from a couple of target portals, messing with automation may not be worth it. Yet, if you have multiple environments with a bunch of portals and targets that need to be discovered and connected, and all of them are more or less similar in terms of configuration, you might find your resort in automating the whole process.

I hope to post some other automation things here, so tune in and check the StarWind blog from time to time.

Please rate this


Read in my feedly

Sent from my iPhone

Beyond The Phoenix Project Audio Series & The 5th Anniversary Edition of The Phoenix Project

Beyond The Phoenix Project Audio Series & The 5th Anniversary Edition of The Phoenix Project
// IT Revolution

Last month, in the IT Revolution Newsletter, I described how The Phoenix Project is celebrating its fifth birthday.

In commemoration, we are releasing something I'm super excited about, titled Beyond The Phoenix Project.

This is nine hours of audio (yes, it grew by two hours since January!) that John Willis and I recorded over the course of the last four months, based on hundreds of hours of research, preparation and rehearsal.

For those of you who don't know, John Willis is one of my co-authors of The DevOps Handbook. Not only is he a good friend of mine, he is without a doubt, one of the original driving forces behind the DevOps movement and one of the co-founders of DevOpsDays as well as the unofficial DevOps historian.

John proposed this project years ago, inspired by our love of Dr. Goldratt's "Beyond The Goal" audio series — this is something I listened to scores of times during the development of The Phoenix Project.

I learned so much during this project — I'm sure even the most ardent DevOps scholars and enthusiasts will learn many surprising and awesome things, just like I did.

Over the next week, I'll share with you some of my favorite highlights and excerpts from the nine different modules of Beyond The Phoenix Project. I hope you enjoy them!



PS: We'll also be releasing the transcripts from Beyond The Phoenix Project — surprisingly, it came out to about 90,000 words. This is almost as long as The DevOps Handbook, with almost as many citations! : )

PPS: Next week, we are also releasing the special 5th Anniversary Edition of The Phoenix Project — it has a shiny, sparkly new cover, a new afterword, and includes a significant portion of The DevOps Handbook. I feel it's a better resource than ever for helping people understand why DevOps is important, and its principles and patterns.

The post Beyond The Phoenix Project Audio Series & The 5th Anniversary Edition of The Phoenix Project appeared first on IT Revolution.


Read in my feedly

Sent from my iPhone

Set Your Sights on Becoming a Citrix Networking expert

Set Your Sights on Becoming a Citrix Networking expert
// Citrix Blogs

It's the year of the expert here at Citrix, which means now is the time to hone your expert level skills and prove your knowledge with a new advanced-level Citrix Certification.

Lucky for you, we've got everything to help get …


Related Stories


Read in my feedly

Sent from my iPhone

5 Ways you can leverage the Linux community for your data center network

5 Ways you can leverage the Linux community for your data center network
// Cumulus Networks Blog

Here at Cumulus, we often talk about the benefits of having an operating system built on Linux (if you need to be re-schooled on the benefits of unifying the stack, head here). But something that possibly goes overlooked, or at least under appreciated, is the value of the Linux community itself. The community is made up of 50,000 or so engineers all passionate about learning, improving and creating code. People like to say that when you go with a Linux operating system, you're "standing on the shoulders of giants," meaning that you don't only have to rely on your inhouse engineering team (even if they're world-class engineers), but rather you're relying on thousands of engineers, including some of the absolute best in the business. Since Cumulus Linux runs on Linux, our customers have this community at their disposal. So why does that really matter? Here are five reasons to consider.

1. Security

The most widely cited benefit of having a community of 50,000 behind you is security. Basically it looks something like this. Let's say you're with a proprietary vendor (*cough* Cisco *cough* Juniper *cough*), and there is a glitch in your latest package installation causing a security vulnerability. Maybe you know about this vulnerability due to functionality issues or because it's public knowledge, so you're sitting around panicking wondering if your network is seconds away from being hacked. Or maybe, in a possibly worse case, you don't even know this vulnerability exists because you can't see the code. You're humming happily to yourself while you close out tickets, and meanwhile, your network has a major gap in its security. In either circumstance, you are 100% dependent on your proprietary vendor knowing about the issue, detecting the cause of the issue, fixing the issue and then sending you a new package containing the fix. Who knows how long your network security was at stake?

With Cumulus Linux, you have the entire Linux community on your engineering team (or at least it may feel that way). In this scenario, being the smarty pants you are, you've built your network with Cumulus Linux. So let's say a bug exists (they happen to the best of us). You might find out about this from a Linux forum, or maybe because you reviewed the code and saw a configuration issue, or maybe someone from the Cumulus team has contacted you. You wonder to yourself how long it will take to fix. Panicking, you start scouring repos, forums and blogs and realize that the Linux community is already on it. Hundreds, maybe thousands of engineers are looking for a way to remediate the issue. Inspired, you start scouring the code as well (might as well join the cause since you have access to the code yourself!). Within hours, the glitch has been found, diagnosed and patched. Cumulus sends you a quick script or package and everything goes back to normal. Quick, easy and painless 🙂

2. Customizability

Linux community benefits also include having an open source code. If you don't like something, you can take it upon yourself to change it! No calls to support required. This may be a benefit to any open source software, but with Linux, you can also take to the community to help you design a solution or even a new feature to customize your data center network exactly to your business requirements and needs. When it comes time to implement the change, you don't have to set up a separate sandbox or container like you would with a proprietary vendor (rhymes with disco). You can go ahead and edit the code yourself, implement the feature, QA and even submit it to the community to adapt into the kernel. Your ideal network design is at your fingertips.

3. Awareness

You may have seen this tactic play out with some of the web-scale giants like LinkedIn, Facebook or Google. Essentially, you can use the Linux community to a) gain an instant user base for your new project or modification, b) leverage the wisdom of the crowd by taking in external contributions and bug fixes and c) cultivate a pool of candidates that are familiarized with your open source operational tooling

For example, a few years ago, LinkedIn developed a product called Kafka, which is a "highly scalable messaging system that plays a critical role as LinkedIn's central data pipeline." LinkedIn released Kafka as an open source ecosystem and let the Linux community run wild. The community was able to analyze, refine and innovate Kafka, and LinkedIn was able to leverage their innovations. When it came time to hire inhouse, LinkedIn had a large candidate pool of engineers that already knew Kafka and had possibly even worked on it.

For companies of all sizes, you can develop any script, feature an application in Linux and submit it to the community for feedback and improvements.

4. Support

Similar to security and awareness, you can also leverage the Linux community for support. Instead of waiting for hours on the phone with a support team, you can head to your favorite forum, mailing list or IRC room and enlist the help of others to solve a problem, code a fix, bake an idea or simply give you a virtual high five. Most of Cumulus Linux is written in Python. So you, and whomever you enlist, can just open the file and start reviewing the code. We like to think this empowers our customers to take their network into their own hands. Whether that be in design or troubleshooting, you have the tools you need to fix it yourself (but of course we're available 24/7 if you need us)!

5. Package-less patches

This last one is a bit more straightforward. At a proprietary vendor (think Mars, but bigger and farther away from Earth), without access to the code, you're left beholden to the vendor — waiting with your hands outstretched for a package for what might amount to a one-line change in a program. With Cumulus Linux, you have access to most of the code and it's in a language that you already know. So, when you're working with our support organization for a fix that is fairly small, instead of waiting for a package in a hot fix scenario, we may edit code right there with you for the fastest resolution to the issue. Of course major fixes always require a package, but you won't incur additional business impact for a fix that should take minutes.

There are countless benefits to having thousands of engineers supporting your network design, operating system and infrastructure. If you're new to Linux and would like to learn more, check out our first white paper in a 5-part Linux series, "What makes up the modern Linux OS?".

The post 5 Ways you can leverage the Linux community for your data center network appeared first on Cumulus Networks Blog.


Read in my feedly

Sent from my iPhone

Automation won’t cost you your job — it could save it

Automation won't cost you your job — it could save it
// Cumulus Networks Blog

If you're a regular reader of our blog, you probably do a lot of professional work with networking, manage an enterprise data center or play around with networks as a hobby (if you don't, close your eyes for just a moment and imagine yourself in a well air-conditioned data center). You also likely know about the day-to-day tasks that maintaining a network requires, and how much time they take out of the day. Or, perhaps you're a director that's trying to resolve the issues your networking team keeps having. Has it ever occurred to you that there might be a better way to tackle these daily problems? Sure, what you're doing now works, but there's so much else you could be doing if the management of these tasks were optimized. That's where network automation solutions can step in and give you more free time than you could have dreamed of. Why automation? Well, let's get into what problems it eliminates and the benefits it brings — you can thank us later!

Problem: A manually configured and operated network

A day in the life of a network engineer includes three layers of regular tasks. At the top, we have troubleshooting operation issues. This takes priority over the others, since issues that occur today require immediate attention versus long-term fixes and fire drills. Next is answering tickets and editing configuration accordingly. These tasks focus on adding new applications and functionality on a per request basis. Again, correcting current problems and completing necessary re-configurations is more crucial than thinking ahead to tomorrow's problems. Finally, the lowest priority is optimizing architecture for future-proofing. While an important part of creating a lasting data center, network optimization falls to the bottom of the priority list since it doesn't involve immediate problems (for now).

Professionals are no strangers to these demands, but have you ever stopped to think about what this repetitive routine is costing you? Symptoms of non-automation-itis include:

  • Intense stress and lack of sleep due to constant troubleshooting and being blamed for issues you didn't create
  • Burnout from the mind-numbing repetition of doing the same simple tasks every day
    Sloppy patches from reactive configurations, as opposed to preemptively dealing with potential errors
  • Bloated costs and broken budgets caused by preventable network downtime, which can take hours to remediate
  • Inability to appropriately scale with the growth of your business
  • Dizziness and uncontrollable hiccuping

If you experience one or more of these side effects (okay, maybe not the last one), talk to your networking vendor today about treating them with automation! Network automation solutions can cure even the most terminal of optimization issues and even improve your network's health.

Solution: What an engineer can accomplish with automation's freedom

Automation gives you a lot more free time (and since we all know time is money, you can expect decent savings as well). With these extra hours, you can accomplish feats like:

Architecture optimization: Without repetitive operations tasks hogging all of your time, you finally have the opportunity to take a good look at your network's architecture and think about how it can be done better. Play around with different configurations. Maybe even look into solving layer 2 complications and implementing layer 3 solutions. Pinpointing the recurrent problem areas in your network and adjusting them will save you from having to deal with inevitable operation issues in the future.

Breaking down bottlenecks: There's no "I" in "team," but there is in "automation!" Now that you've got a moment to look up from your monitor, you can take the time to reach out to fellow employees across different teams. Once your end of the operation starts running smoothly, the positive effects will work their way further down the line and make things easier for others as well. They'll probably find they suddenly have a little more free time! Try communicating with other teams to figure out how you all can better interoperate and get the whole company running more efficiently.

Creating documentation: With these extra hours in your pocket, you have the opportunity to record the work that's been done. Documenting your solutions and innovations is a great way to make sure everyone at the company can access, understand and replicate them (as opposed to directly asking you for an explanation). Plus, posting your work to github or even submitting to the Linux Kernel allows the community to benefit from your genius as well!

Trying out new tools and tech: There's a lot of cool new trends that are taking the data center by storm, from containers to private clouds. Learning to use automation tools like Ansible, Puppet and Chef not only increases your knowledge of automation, but also gives you the free time to catch up on all of the hot new data center practices that can revolutionize your network. Additionally, you can use this time to take advantage of great certification programs to add new abilities to your wheelhouse. Developing these skills can help you remain competitive in the ever-evolving tech world, ensuring job security or even the possibility of career growth.

Ensuring quality: As the saying goes, "measure twice, cut once." It's difficult to keep up with the constant demands of troubleshooting and validation, but network automation solutions make it possible to stay ahead of the game. With more time to devote to quality assurance, you can validate deployments before deploying and be certain that you're rolling out the best.

Treating yourself to better coworkers: When you're working from immediate issue to immediate issue, there's zero time to spend on ensuring the people you're hiring are the best your company can do; you end up hiring based on current needs and desperation rather than long-term potential and how they fit in with the company's mission and environment. Consider using the free time automation provides to keep potential candidates in the interview cycle longer. You'll be able to weed out less viable candidates and recognize where true talent and skill lie. It pays to hire premium engineers. Remember — you could be working with these people for a long time. Having the ability to confirm you're bringing on colleagues who will make your job easier will make your life easier as a result. No need to worry about coworkers whose mistakes reflect poorly on you (or who steal your lunch from the fridge).

Quit taking work home with you: What happens in the office should stay in the office! No one should have to lie awake at night worrying about everything that could go wrong when they step away. Thanks to automation, your network is running as intended with little to no supervision. Sleep easy knowing you won't be woken up in the middle of the night due to a networking issue (that might not be your fault).

Organizational optimization: Since automation gives you more time to optimize your network, you can research and develop more ways to save even more time! As you implement more time-saving techniques, the hours you're NOT using to carry out mind-numbing tasks grow exponentially. Plus, your newfound ability to spread out solutions among cross-functioning teams means that your organization as a whole becomes more efficient. Your decision to automate will earn you the gratitude of your peers (and possibly the recognition of your superiors).

How can Cumulus help you achieve your automation dreams?

If you're looking for a place to begin your journey with network automation solutions, we've got you covered. Cumulus Networks is 100% Linux, so when you're setting up the network, you're not just working with Cumulus — you've got the support of the 50,000+ engineers that make up the Linux community to help you with quicker troubleshooting and patches. No need to sit around and wait for proprietary vendors to fix things for you; take the power into your own hands.

The tech helps keep things running as well. Cumulus Linux is an operating system built to make configurations and deployment simple with its unique CLI, Network Command Line Utility (NCLU). Our telemetry-based fabric validation system NetQ also makes automation a breeze because its preventative nature enables you to reduce errors, and thus downtime, saving you thousands of dollars.

Eager to incorporate automation and improve your network? We've got more information and plenty of resources on our website's network automation solutions page. Head over there and start reaping the benefits of automation!

The post Automation won't cost you your job — it could save it appeared first on Cumulus Networks Blog.


Read in my feedly

Sent from my iPhone

What it means to be an “automation first” enterprise

What it means to be an "automation first" enterprise
// Cumulus Networks Blog

We've previously discussed how automation can give engineers some well-deserved extra free time. So how do those benefits extend to helping the company as a whole? Well, according to TechTarget's article analyzing Gartner's recent report about network innovation, there are some pretty obvious indicators that a company is putting automation first and achieving success. All a business has to do is take advantage of the automation practices used by hyperscale data centers. While it may sound impossible to operate on the same level as cloud giants like Amazon and Facebook, Gartner's report states that there is a remarkable increase in efficiency and agility enterprises mimic from even 1% to 10% of the practices in hyperscale data centers. In this post, we'll discuss what it looks like to be an "automation first" enterprise. And from what we can tell, it looks pretty good!

Reduced costs

Adopting automation doesn't just save you time — it also saves you money. Let's start with the fact that proprietary solutions are incredibly expensive on their own. In addition to steep initial costs, proprietary vendors prevent additional savings by not allowing customers to take advantage of automation tools like Ansible, Puppet and Chef. However, these capabilities are available to you when you switch to open solutions. Having the freedom to utilize these applications creates even more savings. By removing the need for engineers to manually configure everything, automation greatly reduces the possibility of human error bringing your network down. Gartner estimates that "organizations that automate 70% of network configuration changes will cut the number of unplanned outages by more than half when compared to enterprises that automate less than 30% of those changes," and these preventative measures can in turn help you save the thousands of dollars that network downtime costs. In fact, Gartner states that you can reduce expenses by 25% over five years by taking advantage of the automation opportunities that open source networking can offer.

Focus on innovation, not fear

One of the main reasons that enterprises gravitate toward proprietary vendors, despite the many advantages of open source solutions, is that they're more interested in the stability that accompanies the status quo than seeking new and better options. Unfortunately, giving into this mindset hampers innovation because closed networks do not allow you to seek out alternate solutions and options that are tailored to your specific needs. As the TechTarget article points out, "To make use of open source technology, enterprises will have to learn how to manage risk, rather than avoid it at all costs. One way to reduce risk is to segment the network into logical building blocks to contain the growing pains of new technology." (If that last sentence sounds familiar to you, it's probably because our very own Pete Lumbis said a similar thing at Networking Field Day 2017 with a Lego analogy. You can watch his presentation for yourself here.) When enterprises value risk aversion over embracing the latest developments and innovations in technology, they're likely to fall behind due to the inability to scale and adapt with the changing technological environment. Embracing innovations such as automation tools actually helps you avoid the risk of becoming outdated.

Thinking outside the box

The traditional buying model for enterprises involves first choosing the hardware and then following along with the applications can be used with it. But what happens when you reverse this model? When you take the focus off of the hardware and start with applications, you give yourself more options and the ability to make sure your network is customized to your needs, rather than being told by the proprietary vendor what you're allowed to use. The choice and power is in your hands. Once you start reaping the benefits of automation, you'll be glad that you didn't allow vendors to make decisions for you.

Prioritizing people

Of course the technology involved in your network is important, but let's not forget about the priceless value of the talent and dedication provided by the people who make the magic happen. We've talked before about why it's a good idea to spend more on employees and less on technology, and TechTarget backs that ideology, stating "enterprises should shift spending toward retraining network operators and away from buying fully integrated, and often proprietary, networking gear." Once your engineers develop these new skills, you'll find that the risk-aversion mindset you've abandoned gives them the room to innovate and improve your network. Plus, incorporating automation will free up more time for more innovation, so your network can exponentially grow more efficient. You can tell when an enterprise adopts an "automation first" attitude because it's evident in the skills, talent and forward thinking its employees develop.

It's one thing to talk about being "automation first," but what does it look like in practice at a technical level? At Cumulus Networks, we wholeheartedly believe that automation can only make a company better, and we optimize our technology to ensure automating your network is a breeze. Cumulus Linux is a "just Linux" operating system, so we gain all of the benefits of Linux systems since automation tools are geared primarily towards Linux. It works much better than proprietary operating systems, which try to force automation into a closed system like a square peg in a round hole. Cumulus provides the ability to build "workflow" automations, instead of just device automations. Using the same Linux-based automation tools as servers means end-to-end orchestration is possible, where a new server can be deployed and all relevant network configuration is applied.

Searching for a knowledgeable instructor to help you get started with automation? Want to learn more about the differences between automation vs. manual configuration? Watch our how-to video series about automation for expert guidance!

The post What it means to be an "automation first" enterprise appeared first on Cumulus Networks Blog.


Read in my feedly

Sent from my iPhone

CI/CD: What does it mean for the network engineer?

CI/CD: What does it mean for the network engineer?
// Cumulus Networks Blog

The continuous integration/continuous delivery (CI/CD) process is very popular in the DevOps industry. CI/CD creates a more agile software development environment which provides benefits including the faster delivery of applications. As a network engineer, are there any aspects of this I can benefit from to improve network operations and achieve the same goal: design and deploy an agile network that provides customers access to those applications as fast as they are deployed? After all, quick reliable application delivery is only as fast as customers can access it.

This blog post outlines how treating infrastructure as code and implementing a CI/CD workflow can ease the life of a network engineer. It also describes how using Cumulus VX and Cumulus NetQ can simplify this process further.

What does "infrastructure as code" mean?

Generally, it means having all your network node configurations as code that you manage external to the nodes. The program identifies each individual node and renders or produces all the configurations for all the nodes in the network in one step. This also means all configuration changes happen in this code, and the code itself accesses the nodes to deploy the configurations, not the engineer. Configuration deployment can be done automatically, or an engineer can invoke it during an outage window.

Infrastructure as code can be implemented via home-made programs or DevOps/NetDevOps tools such as Ansible, Puppet or Chef. For example, Ansible allows us to represent all the switch and server configurations in the network as a piece of code. Ansible calls the piece of code a playbook. Ansible is also capable of deploying the configurations. Since Cumulus Linux is just Linux, we can use the same modules used by the servers for our switches, unifying the network.

Usually the CI/CD methodology is applied when using infrastructure as code.

What exactly is CI/CD?

CI/CD is a process for an engineer to provide their incremental code changes swiftly as well as create reproducible, testable builds. A plethora of different approaches exist to implement CI/CD, but usually a developer checks out a code branch, makes a change to that code branch and then integrates it back to the master. The new change can be immediately tested via automation (the merge can trigger the validation or the validation can happen at regular intervals). This process helps catch issues early, which subsequently saves time and allows the developers to rapidly provide their code to customers.

CI/CD workflow

Some organizations go one step further by also enabling continuous deployment. Continuous deployment means that as soon as the code passes validation, it is automatically configured on delivery, deployed on the servers, and monitored which can provide time saving and convenience.

Tools such as Jenkins with Github help expedite and automate the CI/CD workflow. We will cover more information on the workflow in a future post.

As a network engineer, why do I care about CI/CD?

As we move from more traditional, monolithic networks into agile web-scale networking, we need a more efficient and reliable way to make changes to the network. Network engineers typically access nodes individually and manually perform CLI commands since all the configs live on each network node. The old method can be tedious, error prone, and may require a longer outage time for changes. CI/CD workflow


Moving into the modern era, we can apply the same CI/CD principles that apply to developers to our configurations – which are now represented as code (e.g. an Ansible playbook). The code is placed on a version controlled repository like github. If we need to make a network configuration change, we would first check out a branch of the known good code. Then we make the required changes to the code and merge it back with the master. We can then validate the changes in a virtual environment before deploying the changes to the production network.

Our new method greatly reduces potential downtime in the production network as seen below.

CI/CD workflow

How can Cumulus VX and NetQ help simplify it even further?

CI/CD involves deployment and validation in a virtual environment prior to production deployment. The validation in a virtual environment greatly reduces the potential for network outage during a change. Cumulus VX is used to set up the virtual environment, as described in this blog post.

To perform automated validation, many engineers write a large number of test cases, which can be extremely tedious. NetQ, along with Cumulus VX, greatly simplifies the validation process in CI/CD by no longer requiring the engineers to write the tests, let NetQ do the test writing for you!

NetQ validation integrates the switch and the host together, so the testing can be done end-to-end. NetQ even integrates containers into the environment.

For example, a engineer may need to write a test case to check all the MTUs in the network and another test case to check all the BGP peers. NetQ has these already done – a few commands below show some MTU mismatches in the network and show a BGP peer is down, caused by a down interface. Note the NetQ commands can be run from anywhere in the virtual network, or directly from your automation tool, the choice is yours.

CI/CD workflow

NetQ show commands can also be used for validation. For example, with one simple netq show clag command, we can see the status of all the CLAG peers in the network.

CI/CD workflow

We can also trace the paths from end-to-end to make sure all our ECMP paths are up and operational.

The CI/CD workflow using Cumulus VX and Cumulus NetQ to provide validation provides much faster, robust network delivery that results in more reliable access to the money making applications, increasing your company's bottom line. As a network engineer, this also means more design time and less outage time spent troubleshooting unforeseen errors.

How can I check this out for myself?

Try it out using Cumulus in the Cloud in a blank slate environment. Cumulus in the Cloud provides an automatic VX environment to test configurations. An Ansible playbook to validate with is located right on github. After spinning up your CITC environment and downloading the playbook to your CITC environment, test it out for yourself. The instructions are right on the github page. You can then insert a few errors into the playbook such as a MTU change or an incorrect BGP peer. Run the playbook with misconfigurations and use NetQ to find the errors. Fix the errors in the playbook, re-run the playbook and watch the errors disappear from NetQ. You will see how easy it is to find configuration and deployment errors with one or two commands, which will save you time. This gives you more time for the fun stuff like designing and optimizing your network.

The post CI/CD: What does it mean for the network engineer? appeared first on Cumulus Networks Blog.


Read in my feedly

Sent from my iPhone

Chef Open Source Community News – February 2018

Chef Open Source Community News – February 2018
// Chef Blog

Here is this month's roundup of news from Chef's open source projects: Chef, Habitat, InSpec, and other miscellaneous tools.


The Chef Client team is busy preparing for the release of Chef 14 in April and trying to squeeze in as many features before we start creating release candidates. If you want to start following along with our alpha releases have a look at the current channel bearing in mind that these builds are still missing major features or could be broken, so don't use them for anything other than testing.

If you are preparing for the end-of-life of Chef 12 on April 30 or interested in what's coming up in Chef 14, we wrote about that earlier in the month in another blog post, with links to upcoming webinars and other resources.

Finally, we released Chef 13.7.16 right after last month's community news update, but unfortunately introduced a regression related to the handling of attributes for folks that manipulate hashes or arrays of attributes inside cookbooks. We're very sorry about this and in the next few days, we'll be releasing a Chef 13.8.x in order to correct this.


The Habitat team has delivered three releases since the last community update. Some valuable features include the ability to create or delete channels in Builder using the hab bldr channel subcommand, the ability to run Habitat as a Windows service, and an easy way to inject secrets in the build studio using hab studio secret.

On the integrations front, we have expanded our ability to export Habitat packages and run them in different runtime environments. First up, we now allow running the supervisor unprivileged in order to allow you to export and run your packages in Red Hat OpenShift. And just in time for the Helm Summit, happening as we speak in sunny rainy snowy Portland, Oregon, we announced a Helm chart exporter that will work with the previously-announced Kubernetes operator. Have a look at our blog post on that topic.


Our big news around InSpec was Tuesday's launch of InSpec 2.0, whose main feature is the ability to now define compliance profiles to test clouds like AWS and Microsoft Azure. InSpec 2.0 also comes with many more out-of-the-box resources compared to InSpec 1.0 as well as drastically improved performance. Check out our release announcement or get started with the new cloud resources right away.


Chef maintains a myriad of other open-source projects such as Test Kitchen and Foodcritic. Test Kitchen and its drivers got some useful features in the last month, chiefly the ability for Test Kitchen to now use Hyper-V as a hypervisor for your cookbook testing. Just this week we also added the ability to use Differencing Disks in Hyper-V to speed things up. We also updated the Test Kitchen VMware vRealize Automation (vRA) driver to be able to select images by name rather than UUID, a significant quality of life improvement. And finally, Test Kitchen gained the ability to use HTTP proxies to SSH to machines.


Notable events that we were at since the last update included CfgMgmtCamp 2018 in Gent, Belgium, our first Chef Community Summit in Berlin, as well as the Helm Summit. Come say hi if you're at DevOpsDays Charlotte today or tomorrow where Nathen Harvey is giving a workshop on Habitat this afternoon. And check out our events website to see where Chef will be later in February and March.


Although February's a short month, we've already had a busy one so far. There's only a few months to go until ChefConf 2018, and we'll be announcing the list of speakers soon. Meanwhile, you can register for the conference for less than $1000 which gets you access not only to the full conference but one workshop, a certification exam, all social activities as well as our hackday. We're looking forward to welcoming you in Chicago in May, and thanks again for using Chef.

The post Chef Open Source Community News – February 2018 appeared first on Chef Blog.


Read in my feedly

Sent from my iPhone

ChefConf Diversity Scholarships

ChefConf Diversity Scholarships
// Chef Blog

ChefConf 2018 will be our 7th annual event. This year we'll be in Chicago in May, celebrating all the great accomplishments of the Chef community with deep dish pizza. And an amazing lineup of talks and events!

Our theme this year is "Do Change" and one of the things we are hoping to change is the overwhelming lack of diversity in the tech industry. It's been much in the news over the past couple of years, and we are committed to doing our part. Unfortunately, change hasn't come yet, so we are again offering our ChefConf Diversity Scholarships for 2018 to help those folks who wouldn't otherwise be able to join us.

This is an opportunity for folks in underrepresented groups to come and participate. Hang out with us for the week! The scholarships include all of our keynote sessions, breakouts, social events, meals, and a seat in a Chef Certification Exam. Everything but travel and hotel.

Network with community members from hundreds of companies, mix and mingle with Chef employees and #cheffriends from all over the world, and learn about some of the innovative projects and solutions being created with Chef, InSpec, and Habitat.

The deadline is March 12, so don't wait to apply!

The post ChefConf Diversity Scholarships appeared first on Chef Blog.


Read in my feedly

Sent from my iPhone

Habitat and Helm Exporter

Habitat and Helm Exporter
// Chef Blog

A group of Habitat core team members is at Helm Summit in beautiful (but surprisingly frosty) Portland this week, to learn more about how Helm contributors are continuing to automate application deployment and management to Kubernetes clusters.

This week we also released our new Habitat and Helm exporter and we are excited to share this with the Helm community. Check it out in this blog on the project site and let us know how it works for you!

About Helm

Helm is an open source package manager for Kubernetes that is maintained by the CNCF. It allows you to define, install, and upgrade even very complex Kubernetes applications — and you can run multiple versions of the same service within the cluster, which is very helpful when you're managing different releases.

About Habitat

Habitat allows you to automate how you build your application, so that you explicitly declare build and runtime dependencies, and then create a Habitat artifact that only includes your code and its runtime dependencies. Once it's deployed, your artifact is also built with application lifecycle hooks that tell it how to behave. Habitat will rebuild your application as your update your application's code in your source control tool of choice, or if an underlying dependency has a new version.

Habitat also has exporters, which run after you build your application, and export your application to various formats so you can run the exact same application, bundled with its runtime behavior, in many different environments.

As of Habitat's 0.54.0 release, you can use Habitat with the Habitat Operator for Kubernetes (which is now at 0.5.0 and allows you to automate how you manage your applications on your Kubernetes cluster with Habitat) and export your application to run in Helm charts. Check it out, let us know how it goes, and Happy Helming!

Helpful Links

The post Habitat and Helm Exporter appeared first on Chef Blog.


Read in my feedly

Sent from my iPhone

InSpec 2.0 Cloud Resources Mini-Tutorial

InSpec 2.0 Cloud Resources Mini-Tutorial
// Chef Blog

InSpec 2 introduces the ability to test cloud resources for compliance in addition to the system and application-level resources that previously appeared in InSpec 1. In this tutorial, we'll use InSpec to write a very simple control to check attributes on a virtual machine in Amazon Web Services (AWS).

For the purposes of this tutorial, we're going to assume you already have an AWS account and that you're familiar with how to create virtual machines using the AWS console. We also assume you've used the AWS command-line tools before and have Identity and Access Manager (IAM) credentials set up. If neither of these is the case, please consult the relevant AWS tutorial before continuing further.

Let's go to the AWS console and launch a new EC2 instance that we can use for our test. We'll create one using the Amazon Linux AMI:

On the next screen, let's choose to make the instance size t2.nano in order to keep our costs low for this exercise.

Don't click Review and Launch yet. We're going to modify a few parameters before we launch the instance, so click Next: Configure Instance Details first.

On the next screen, let's make sure this instance doesn't have a public IP address, so set Assign Public IP to Disable:

Click Next: Add Storage but skip the storage configuration screen by clicking Next: Add Tags. Let's add a Name tag by following the link to do so and name it "InSpec test server".

Great. We have configured enough of our instance to click Review and Launch.

Once you've launched your instance and it's fully provisioned, it's time to use InSpec to write a cloud compliance rule against it. In this tutorial, we're going to use the InSpec shell to make sure we have our syntax right. We could then take our code and put it in a full InSpec profile later if we wanted to.

If you don't already have InSpec installed, please visit the downloads page to get it. You'll want to make sure you are using InSpec 2.0 or later.

Now let's start the InSpec shell using the AWS driver:

$ inspec shell -t aws://

If you started your EC2 virtual machine in a region that's different than the default one specified in your AWS CLI configuration file (~/.aws/config), you'll want to specify the right region, for example:

$ inspec shell -t aws://us-east-2

The shell will start up and give you a prompt like this:

Welcome to the interactive InSpec Shell  To find out how to use it, type: help  You are currently running on:      OS platform: aws      OS family: cloud      OS release: aws-sdk-v2.10.133  inspec>

Let's write a simple control to make sure the instance is running. Enter this directly into the shell:

describe aws_ec2_instance(name: 'InSpec test server') do    its('state') { should cmp 'running' }  end

After you type the final end, InSpec will evaluate the test:

Profile: inspec-shell  Version: (not specified)    EC2 Instance InSpec test server         state should cmp == "running"  Test Summary: 1 successful, 0 failures, 0 skipped

Great! If you got an error, check that you named the server in the EC2 console the same as what you put in the test.

We can now augment our test with some of the other attributes we configured the instance with. For example, we can make sure the instance does not have a public IP address:

describe aws_ec2_instance(name: 'InSpec test server') do    its('public_ip_address') { should be_nil }  end

We could also check to see that the instance is using an approved Amazon Machine Image (AMI):

describe aws_ec2_instance(name: 'InSpec test server') do    its('image_id') { should cmp 'ami-f63b1193' }  end

(The actual image ID for the Amazon Linux image will vary by region so you may get a test failure.)

For more information on the types of parameters that can be tested, please consult the InSpec documentation.

Before we leave the shell and finish this tutorial, let's have a look at all the other cloud resources you can use. Type

help resources

into the InSpec shell and look at all resources starting with aws for AWS resources, and azure for Microsoft Azure resources. You can type help followed by the resource name into the shell for a brief description of how to use it, or consult the InSpec documentation for a longer description.

Finally, exit the shell by typing quit and pressing Enter. (Don't forget to terminate your AWS EC2 instance once you're done experimenting!)

We hope this brief tutorial has helped illustrate the cloud compliance resources available in InSpec 2 and how to get started using them. We can't wait to see the profiles you create using this new feature. Thanks again for using InSpec!

The post InSpec 2.0 Cloud Resources Mini-Tutorial appeared first on Chef Blog.


Read in my feedly

Sent from my iPhone

Announcing InSpec 2.0

Announcing InSpec 2.0
// Chef Blog

We are delighted to announce the availability of InSpec 2.0, the newest version of Chef's open-source project for compliance automation. InSpec helps you express security and compliance requirements as code and incorporate it directly into the delivery process, eliminating ambiguity and manual processes to help you ship faster while remaining secure.

What's New in InSpec 2.0

InSpec 2.0's newest feature is the ability to test cloud resources for compliance, not just machines, by connecting directly to cloud provider APIs. Today we are launching with support for Amazon Web Services (AWS) and Microsoft Azure, with more to come. For example, here is how you can use InSpec to check for insecure AWS S3 buckets, which is a common security problem that has been in the news recently:

describe aws_s3_bucket(bucket_name: 'my_secret_files') do    it { should exist }    it { should_not be_public }  end

We can also write a similar rule for an Azure storage group containing publicly-accessible blobs to make sure it follows best practices:

describe azure_generic_resource(group_name: 'my_secret_storage', type: 'Microsoft.Storage/storageAccounts') do    its('properties.encryption.keySource') { should cmp 'Microsoft.Storage' }    its('') { should be true }    its('properties.supportsHttpsTrafficOnly') { should be true }  end

Get Started with InSpec 2.0

To get started with the new compliance features in InSpec 2.0, please see this brief tutorial that shows you how to check several aspects of an Amazon virtual machine instance. You can also look at InSpec's documentation which illustrates all the cloud resources available for testing. Shortly, we will have additional tracks on Learn Chef Rally with extended tutorials for cloud resources on both AWS and Microsoft Azure.

InSpec 2.0 also includes over 30 new resources to help you test common system and application configurations for conformance. You can now natively write InSpec rules for checks as diverse as SQL database configurations, webserver (Apache/IIS/NGINX) configurations, Docker images, and much more.

Thanks to our contributors 

We'd like to thank all of our open-source InSpec community members for helping to make this release amazing, particularly our development partners at D2L and MITRE, as well as our InSpec engineering team here at Chef. Thank you for using InSpec and we hope you enjoy the new release.

Learn more

  • Register to attend our live webinar, "Augment your Audits with InSpec 2.0″, on Wednesday, March 21st at 11:00 AM PT. See a live demo of what's new in InSpec and learn how to kickstart your continuous compliance journey.
  • Want a taste of InSpec right away? Our preconfigured, ready-to-go environments on Learn Chef Rally will help you explore how it works in minutes.
  • Sign up for our InSpec Jumpstart workshop at ChefConf. Learn through hands-on labs how to detect and correct with InSpec to ensure you have a secure and compliant infrastructure.

The post Announcing InSpec 2.0 appeared first on Chef Blog.


Read in my feedly

Sent from my iPhone