Sunday, August 16, 2015

Buildpacks in Lattice [feedly]

Buildpacks in Lattice
http://www.chipchilders.com/blog/2015/8/12/buildpacks-in-lattice.html

-- via my feedly.com reader

Lattice v0.3.0 just shipped (if you don't know much about Lattice, go to Lattice.cf), and for me the big news was the newly added support for Cloud Foundry Buildpacks within a Lattice cluster.

The goal of Lattice is to get a portion of the larger Cloud Foundry platform repackaged into an easily consumable system that makes getting started with the Cloud Foundry technologies straight forward for individuals and small teams. If you want to understand this a little better, watch James Bayer and Colin Humphreys talk about Lattice at the 2015 CF Summit North America.

In previous releases, Lattice was limited to only running completed Docker images within the cluster. Now that we have Buildpack support, Lattice is really able to give you a much better taste of what it means to use Cloud Foundry. Have some code? Just push it to Lattice and let it do the work to build the container for you.

Here's how to use it:

Setup

For this walkthrough, I'll be using the Vagrant box version of Lattice (using VirtualBox). These instructions assume v0.3.0 of both Lattice and the ltc command line tool. The same process should work for future versions of Lattice, we well as different deployment options (like one of the Terraform configurations for different IaaS environments).

You can grab the ltc cli tool from the bottom of the release notes.

To get the latest 

>  git clone https://github.com/cloudfoundry-incubator/lattice.git  > cd lattice  > git checkout v0.3.0  > vagrant up --provider virtualbox  

This should result in a running box with the Lattice system initialized. If it worked correctly, you'll see:

==> default: Lattice is now installed and running.  ==> default: You may target it using: ltc target 192.168.11.11.xip.io  

By default, the Lattice box environment will be unauthenticated so it's easy to get ltc to target the cluster:

> ltc target 192.168.11.11.xip.io  Blob store is targeted.  Api Location Set  

I've run into issues where xip.io doesn't resolve via some DNS servers, so if you have that issue consider using another DNS server like Google's 8.8.8.8 address.

At this point, you should have a functional Lattice system and ltc should be able to talk to it. Give it a test by running 'ltc list'.

> ltc list  ------------------------------= Apps =-------------------------------  No apps to display.    ------------------------------= Tasks =------------------------------  No tasks to display.  

Get a Sample App

Now we're ready to find some source code that we want to run within Lattice. Since Lattice is specifically designed to be a BYOS (bring your own services), my example will not require any database or other backend services. It's a simple Python Flask based web app. Obviously it's a trivial Hello World example, but you can easily extrapolate and see how this would be useful for non-trivial systems.

The sample application we'll be using can be found on GitHub at https://github.com/chipchilders/sample-python. Go ahead and clone the repo now (from a working directory that's not within the lattice project's directory):

> cd ~  > git clone https://github.com/chipchilders/sample-python.git  > cd sample-python  

There are only three files in the repo:

 
  • Procfile - the file that helps Flask startup
  • requirements.txt - the list of modules needed to run the app, in this case it's only Flask
  • hello.py - the basic web app that we'll be hosting within Lattice
 

The contents of hello.py are pretty straight forward. The file creates a route for '/' that returns the string "Hello World! Im an instance number X", where X is the Lattice instance number (as retrieved from the app's environment variables).

from flask import Flask  import os    app = Flask(__name__)    cfindex = os.getenv("INSTANCE_INDEX")    @app.route('/')  def hello_world():      return 'Hello World! I am instance number ' + str(cfindex)    if __name__ == '__main__':      app.run(host='0.0.0.0', port=8080)  

My example assumes that 8080 is a reasonable listening port, since that's the default port that Lattice expects from containers. There's lots of flexibility here, but defaults will do for the example.

Build the Droplet

Lattice uses the same terminology as Cloud Foundry for the container images created via buildpacks: Droplets. Droplets are the result of taking code from the user and running it through the staging process.

Step one is to do a build of the droplet:

> ltc build-droplet sample-python https://github.com/cloudfoundry/python-buildpack  

The command tells ltc to build the local directory into a droplet, naming it "sample-python" and using the cloudfoundry/python-buildpack. When you want to do something in other languages, the same process applies. Just be sure to specify an appropriate buildpack.

If everything is working correctly, you'll see Lattice stage the code and build a Droplet image:

Submitted build of sample-python  08/12 14:52:58.93 [BUILD|0] Successfully created container  08/12 14:53:07.60 [BUILD|0] -------> Buildpack version 1.5.0  08/12 14:53:07.67 [BUILD|0] -----> Installing runtime (python-2.7.10)  08/12 14:53:15.09 [BUILD|0] -----> Installing dependencies with pip  08/12 14:53:15.44 [BUILD|0]        Collecting Flask (from -r requirements.txt (line 1))  08/12 14:53:15.60 [BUILD|0]          Downloading Flask-0.10.1.tar.gz (544kB)  08/12 14:53:15.96 [BUILD|0]        Collecting Werkzeug>=0.7 (from Flask->-r requirements.txt (line 1))  08/12 14:53:16.04 [BUILD|0]          Downloading Werkzeug-0.10.4-py2.py3-none-any.whl (293kB)  08/12 14:53:16.11 [BUILD|0]        Collecting Jinja2>=2.4 (from Flask->-r requirements.txt (line 1))  08/12 14:53:16.18 [BUILD|0]          Downloading Jinja2-2.8-py2.py3-none-any.whl (263kB)  08/12 14:53:16.25 [BUILD|0]        Collecting itsdangerous>=0.21 (from Flask->-r requirements.txt (line 1))  08/12 14:53:16.31 [BUILD|0]          Downloading itsdangerous-0.24.tar.gz (46kB)  08/12 14:53:16.46 [BUILD|0]        Collecting MarkupSafe (from Jinja2>=2.4->Flask->-r requirements.txt (line 1))  08/12 14:53:16.53 [BUILD|0]          Downloading MarkupSafe-0.23.tar.gz  08/12 14:53:16.66 [BUILD|0]        Installing collected packages: Werkzeug, MarkupSafe, Jinja2, itsdangerous, Flask  08/12 14:53:16.79 [BUILD|0]          Running setup.py install for MarkupSafe  08/12 14:53:17.50 [BUILD|0]          Running setup.py install for itsdangerous  08/12 14:53:17.66 [BUILD|0]          Running setup.py install for Flask  08/12 14:53:18.00 [BUILD|0]        Successfully installed Flask-0.10.1 Jinja2-2.8 MarkupSafe-0.23 Werkzeug-0.10.4 itsdangerous-0.24  08/12 14:53:37.28 [BUILD|0] Exit status 0  08/12 14:53:37.51 [BUILD|0] Uploaded /tmp/droplet to http://192.168.11.11.xip.io:8444/blobs/sample-python/droplet.tgz.  08/12 14:53:37.52 [BUILD|0] Exit status 0  08/12 14:53:37.55 [BUILD|0] Uploaded /tmp/result.json to http://192.168.11.11.xip.io:8444/blobs/sample-python/result.json.  08/12 14:53:37.56 [BUILD|0] Exit status 0  08/12 14:53:37.60 [BUILD|0] Deleted http://192.168.11.11.xip.io:8444/blobs/sample-python/bits.zip.  08/12 14:53:37.61 [BUILD|0] Exit status 0  Build completed  

To confirm that the droplet is registered in the cluster, you can run "ltc list-droplets" (or my favorite shorthand "ltc lsd").

> ltc lsd  Droplet		Created At		Size  sample-python	08/12 18:53:37.00	29.5M  

Next, we will launch one instance with the "ltc launch-droplet" comment. You'll need to specify a name for the application (yes, multiple apps can be launched from the same droplet), as well as the name of the droplet to launch from. I used my imagination and named my app "test" in the example below:

> ltc launch-droplet test sample-python  No port specified. Defaulting to 8080.  Creating App: test  .08/12 15:04:55.53 [APP|0] Successfully created container  ....08/12 15:05:00.01 [APP|0]  * Running on http://0.0.0.0:8080/ (Press CTRL+C to quit)  .08/12 15:05:00.39 [HEALTH|0] healthcheck passed  08/12 15:05:00.39 [HEALTH|0] Exit status 0    test is now running.  App is reachable at:  http://test.192.168.11.11.xip.io  

And there we go! You should be able to see the app working by hitting http://test.192.168.11.11.xip.io. You can also now take advantage of the other features of Lattice, like scaling up / down (ltc scale), automatic health management and log aggregation.

Let's play with Lattice

To see the app list:

> ltc list  ------------------------------= Apps =-------------------------------  App Name	Instances	DiskMB		MemoryMB	Route  test		1/1		0		128		test.192.168.11.11.xip.io, test-8080.192.168.11.11.xip.io => 8080    ------------------------------= Tasks =------------------------------  No tasks to display.  

To see the details of the "test" app:

> ltc status test  ==========================================================================================        test  ------------------------------------------------------------------------------------------  Instances	1/1  Start Timeout	0  DiskMB		0  MemoryMB	128  CPUWeight	100  Ports		8080  Routes		test.192.168.11.11.xip.io => 8080  		test-8080.192.168.11.11.xip.io => 8080  Annotation	{"droplet_source":{"host":"192.168.11.11.xip.io","port":"8444","droplet_name":"sample-python"}}  ------------------------------------------------------------------------------------------  Environment    MEMORY_LIMIT="128M"  PROCESS_GUID="test"  PORT="8080"    ==========================================================================================        Instance 0  [RUNNING]  ------------------------------------------------------------------------------------------  InstanceGuid	a4aac9d3-fba5-453b-71ba-ec2004a17138  Cell ID		cell-01  Ip		192.168.11.11  Port Mapping	60004:8080  Uptime		3m57s  Crash Count 	0  CPU 		0.12%  Memory 		13.5M  ------------------------------------------------------------------------------------------  

Scaling up or down:

> ltc scale test 5  Scaling test to 5 instances  .....................  App Scaled Successfully  

Now you should notice when you hit the app's web page that the index will jump around between the 5 instances on each refresh.

Also, the "ltc list" command should now show 5 instances running, and "ltc visualize" should show that there are 5 containers running on your cell (or cells if you deployed a Lattice cluster).

> ltc list  ------------------------------= Apps =-------------------------------  App Name	Instances	DiskMB		MemoryMB	Route  test		5/5		0		128		test.192.168.11.11.xip.io, test-8080.192.168.11.11.xip.io => 8080    ------------------------------= Tasks =------------------------------  No tasks to display.    > ltc visualize  Distribution  cell-01: •••••  

And if you want fancy visualizations of your Lattice environment, check out xray.cf from the kind folks at Pivotal. It will connect to your Lattice environment from your browser, so it even works with your local VirtualBox Lattice instance.

Now let's break something!

Lattice includes both health management (keeping it's promises) as well as great logging visibility into the cluster's operations. To test this out, we can mess with the cluster by killing processes and watch as Lattice heals around our malevolence.

To start, open two terminal windows. In one, we'll use "vagrant ssh" to cause some damage. In the other, we'll watch as Lattice responds.

Get started by running "ltc logs test" in the first terminal to start watching the logs for your test application:

> ltc logs test    08/12 15:24:32.79 [HEALTH|0] healthcheck passed  08/12 15:24:32.79 [HEALTH|0] Exit status 0  ...  

Now in the other terminal, use "vagrant ssh" to log into the Vagrant box so we can do some damage:

> vagrant ssh  Welcome to Ubuntu 14.04.3 LTS (GNU/Linux 3.16.0-30-generic x86_64)     * Documentation:  https://help.ubuntu.com/  vagrant@ubuntu-trusty-64:~$ ps -ef | grep iodaemon | grep -v grep | awk '{print $2}'  13469  14531  14580  14636  14665  

That's the list of PIDs for the 5 instances of the Lattice app. Go ahead and kill -9 one or more of them!

vagrant@ubuntu-trusty-64:~$ sudo kill -9 13469 14531  

If you go over to the terminal window showing logs, you will see that Lattice will notice the issue and quickly restart new app instances within the cluster automatically.

08/12 15:28:33.13 [HEALTH|0] healthcheck passed  08/12 15:28:33.14 [HEALTH|0] Exit status 0  08/12 15:28:38.50 [HEALTH|4] healthcheck passed  08/12 15:28:38.51 [HEALTH|4] Exit status 0  08/12 15:28:40.65 [APP|0] Creating container  08/12 15:28:40.66 [APP|4] Creating container  08/12 15:28:41.68 [APP|0] Successfully created container  08/12 15:28:42.34 [HEALTH|1] healthcheck passed  08/12 15:28:42.34 [HEALTH|1] Exit status 0  08/12 15:28:42.80 [APP|4] Successfully created container  08/12 15:28:45.47 [HEALTH|3] healthcheck passed  08/12 15:28:45.56 [HEALTH|3] Exit status 0  08/12 15:28:45.73 [HEALTH|2] healthcheck passed  08/12 15:28:45.81 [HEALTH|2] Exit status 0  08/12 15:28:48.97 [HEALTH|0] healthcheck failed  08/12 15:28:48.99 [HEALTH|0] Exit status 1  08/12 15:28:49.08 [APP|0]  * Running on http://0.0.0.0:8080/ (Press CTRL+C to quit)  08/12 15:28:49.55 [HEALTH|0] healthcheck passed  08/12 15:28:49.58 [HEALTH|0] Exit status 0  08/12 15:28:50.87 [APP|4]  * Running on http://0.0.0.0:8080/ (Press CTRL+C to quit)  08/12 15:28:51.03 [HEALTH|4] healthcheck passed  08/12 15:28:51.04 [HEALTH|4] Exit status 0  

So there go... All the fun of Lattice, now with buildpacks!