Postmortem: ohai & mixlib-shellout gem release issues [feedly]

----
Postmortem: ohai & mixlib-shellout gem release issues
// Chef Blog

Hey chefs,

On 2014/12/01 we have released a few gems in order to prepare for upcoming Chef Client 12.0.0 release. During these releases we've caused 2 distruptions:

1. ohai 7.6.0 release broke gem installation of chef 11.16.4 when using an older version of rubygems.
2. Yanking mixlib-shellout 1.6.0 release from rubygems.org caused issues in projects with Gemfile.lock files that are pointing to this version.

We have resolved the first issue by yanking ohai 7.6.0 from rubygems.org. The resolution to the second issue is to update the mixlib-shellout to version 1.6.1 by running bundle update mixlib-shellout.

This post summarizes the details of these issues and the fixes that are in flight. We will have a public postmortem on 2014/12/03 at 13:00 PT to determine any further prevention mechanisms and corrective actions. We will tweet the link for the hangout and youtube livestream for folks to join or watch when we are getting started.

"gem install chef" temporarily broken for 11.16.4

Huge thanks to migurski for reporting this issue. You can read the details of the issue here. As migurski pointed out the gem dependencies of recent releases of chef, ohai and mixlib-shellout are as follows:

chef 11.16.4 depends on mixlib-shellout ~> 1.4 chef 11.16.4 depends on ohai ~> 7.4 ohai 7.4.0 depends on mixlib-shellout ~> 1.2 ohai 7.6.0 depends on mixlib-shellout < 3.0, >= 2.0.0.rc.0

In addition to this mixlib-shellout 2.0.0 is not backwards compatible with chef 11.16.4 due to a change around encoding environment variables.

These constraints caused gem install chef command to fail with earlier versions of rubygems since it was not able to find out a solution for the dependency tree even though one exists with using ohai 7.4.0 & mixlib-shellout 1.6.1. This issue was not seen on the newer versions of rubygems

Amongst the many possible resolutions to this issue we have selected to yank ohai 7.6.0 from rubygems.org and decided to release ohai 8.0.0 to go with chef 12.0.0. The reason to select this choice was to ensure that we don't leave any versions of our gems out there which can create issues with the dependency resolution logic of older rubygems.

This resolution also keeps the dependencies clear in between major versions of chef releases. chef 12.X will depend on ohai 8.X and mixlib-shellout 2.X where chef 11.X depends on ohai 7.X and mixlib-shellout 1.X.

The next steps remaining in our resolution are:

• Release ohai 8.0.0
• Release chef 12.0.0 with dependencies mixlib-shellout ~> 2.0 & ohai ~> 8.0

Existing release, chef 11.16.4, will continue to work with existing release of ohai 7.4.0 and mixlib-shellout 1.6.1.

Broken Gemfile.lock files with mixlib-shellout 1.6.0

One of the frequently reported issues for mixlib-shellout 1.6.0 has been the fact that it is broken on Ruby 1.8.7. This was caused by a change that went into mixlib-shellout that wasn't supported in Ruby 1.8.7. Even though we had this change in mixlib-shellout we didn't update the required_ruby_version setting for mixlib-shellout.

In order to recover from this issue we've yanked mixlib-shellout 1.6.0 from rubygems.org and released mixlib-shellout 1.6.1 with only change of including s.required_ruby_version = ">= 1.9.3" in the gemspec of mixlib-shellout. This made sure that in the future we will not have any failures in mixlib-shellout due to the incompatible Ruby 1.8.7 logic.

However, since mixlib-shellout 1.6.0 has been out for a while (since October 7, 2014), it made its way into many Gemfile.lock files. Once this version was yanked, attempts to install this version of mixlib-shellout using already existing Gemfile.lock files started failing. This impacted all the build projects of Chef Software Inc and potentially other external projects.

We were not able to reserve this issue since rubygems does not allow re-pushing an already yanked gems with good reasons. Workaround for this issue remains to be to update the mixlib-shellout version in the gem bundle of the impacted projects with this command:

$ bundle update mixlib-shellout

Postmortem

Join us on 2014/12/03 at 13:00 PT in our postmortem to talk about any additional corrective actions in order not to run into this issue in the future. We will tweet the link for the hangout and youtube livestream for folks to join or watch when we are getting started.

Thanks for your help on getting these issues resolved and your patience while they are sorted out. Feel free to reach us with any questions regarding this in our mailing lists chef and chef-dev or on Github.

----

Shared via my feedly reader

Sent from my iPhone