Kubernetes the Hard Way Made Easier

I recently went on a bit of a quest to teach myself Kubernetes so I could help a couple of software development teams improve their Software Development Lifecycle. I’m a strong believer in lean software development with a strong focus on getting rid of waste. A good way of doing this is putting the right tools in place which kind off enforces lean ways of working. A good example is Continuous Integration or serverless architecture.  You put it in place and when the team starts using it waste (manual installs or setting up and maintaining Jenkins) disappears.

However, all this comes with an (upfront) cost. Either in effort spend beforehand (setting up Jenkins with a scalable cluster of build agents), paying a 3rd party to maintain a part of your development pipeline or a big upfront learning curve (paid in time).

Welcome to Kubernetes. Once it’s up and running it’s a fantastic tool to run an orchestrated  microservices architecture. But setting it up properly, for example, with high availability in multiple AWS availability zones with multiple subdomains, it is not that that easy.

So if you want to set it up yourself and not use a third party ready available cluster, I thoroughly recommend following the Kubernetes the Hard Way tutorial. Not necessarily to setup your production cluster (e.g. it’s not written for high availability in multiple zones). But going through the steps one by one will give you a good basic understanding of the internals of Kubernetes and help you better understand future problems.

For people who are already familiar with networking and certificate generation  I created some Terraform script to setup the network and some bash scripts to guide you through the certificate part. This way you can dive straight into the Kubernetes chapters. The scripts can be found on my Github account in the kubernetes-terraform repository.

You will discover that setting up a Kubernetes cluster is not that easy because there are a lot of moving parts. So if you want to do this I highly recommend Kops, a tool build by the Kubernetes team, and only recently come out of alpha. It helps you setup Kubernetes and  does a lot of work for you. On top of that; the people on the forums are very helpful in case you run into trouble.

 

 

SCTP in Go

Rejoice! There seem to be some interest in my SCTP in Go coding project. After reading some positive comments on Reddit I decided to pick up the work where I left it.

The current state of the project is a bare minimum implementation of the protocol in Go 1.6 in my Github account together with some examples. My first task will be to get it working in Go 1.7.

Here a small preview of a SCTP client written in Go:

package main
import (
	"log"
	"os"
	"net"
)

func main() {

	saddr := "127.0.0.1:4242"
	addr, err := net.ResolveSCTPAddr("sctp", saddr)
	if err != nil {
		println(err)
		os.Exit(-1)
	}

	conn, err := net.DialSCTP("sctp", nil, addr)
	if err != nil {
		println("Error listening " + err.Error())
		os.Exit(-1)
	}

	defer conn.Close()

	var message = "hello"
	bmessage := []byte(message)

	_, err = conn.WriteTo(bmessage, addr)
	if err != nil {
		log.Printf("WriteTo error: %v", err)
	}

}

 

More info about SCTP in Go can be found in my paper I published for my talk at the 2013 AsiaBSD conference or a video recording of the talk I gave at the 2012 EuroBSD conference.

My EuroBSD 2012 talk: SCTP in Go.

Just finished my talk at EuroBSD 2012 about my work extending the Go network library with SCTP. Slides are available here.

On October 21, 2012October 21, 2012 By cyberroadieIn Uncategorized3 Comments

How to test a JPA repository

Here is a typical (Hibernate) JPA repository for saving an entity, a Cookie in this case:

@Entity
@Table(name = "cookie")
public class Cookie {

@Id
 @GeneratedValue(strategy = GenerationType.AUTO)
 private Integer id;

@Temporal(TemporalType.TIMESTAMP)
 @Column(name = "last_modified")
 private Date lastModified;

}

@Repository
public class CookieRepository {

@PersistenceContext
 EntityManager em;

@Autowired
 TimeService timeService;

@Transactional
 public Cookie createEntity(Cookie cookie) {
 cookie.setLastModified(timeService.getTime().toDate());
 em.persist(cookie);
 return cookie;
 }
}

Writing a unit test for this is not as straightforward as you might think. When a new cookie is created the id is null and only after it gets persisted in the database it gets assigned a value generated by the database.

The EntityManager persist method does not return a value, so how can we mock the entity manager _and_ set the id of the cookie?

Here is the answer:

@RunWith(MockitoJUnitRunner.class)
public class CookieRepositoryTest {

@Mock
 EntityManager em;

@Mock
 TimeService timeService;

@InjectMocks
 CookieRepository underTest = new CookieRepository();

@Test
 public void testCreateEntity() throws Exception {
 Cookie newCookie = new Cookie();

when(timeService.getTime()).thenReturn(new DateTime(DateTimeZone.UTC));

doAnswer(new Answer<Cookie>() {
 @Override
 public Cookie answer(InvocationOnMock invocationOnMock) throws Throwable {
 Object[] args = invocationOnMock.getArguments();
 Cookie cookie = (Cookie) args[0];
 cookie.setId(1);
 return null;
 }

}).when(em).persist(any(Cookie.class));

Cookie persistedCookie = underTest.createEntity(newCookie);
 assertNotNull(persistedCookie.getId());
 }

}

Mockito comes with the doAnswer method, specifically designed for dealing with void methods. With this method you can catch the input of the void method, persist() in this case, manipulate the object which than can be asserted for correctness later in the test.

A Scala Fractal

As a follow up from my previous Fern Fractal written in Go, here the same written in Scala;
The fractal is a Barnsley Fern and is an example of an iterated function system.

import java.awt.{Color, Graphics2D, Dimension}
import swing.{SimpleSwingApplication, Panel, Frame}
import util.Random
class FernFractalFrame(transformFunction: (Double, Double) => (Double, Double), val width: Int, val height: Int, val max: Int) extends Frame {

contents = new Panel {
 preferredSize = new Dimension(width, height)
 opaque = true

override def paint(g: Graphics2D) {
 g.setBackground(new Color(0, 0, 0))
 g.setColor(Color.GREEN)
 g.drawLine(height / 2, width / 2, height / 2, width / 2)
 drawFern((0, 1), max, g)
 }

def paintPoint(p: (Double, Double), g: Graphics2D) = {
 val scale = height / 11
 val y = (height - 25) - (scale * p._2)
 val x = (width / 2) + (scale * p._1)
 g.drawLine(x.toInt, y.toInt, x.toInt, y.toInt)
 }

def drawFern(p: (Double, Double), max: Int, g: Graphics2D) {
 paintPoint(p, g)
 repaint()
 if (max != 0)
 drawFern(transformFunction(p._1, p._2), max - 1, g)
 }
 }
}

object FernFractal extends SimpleSwingApplication {

object TransformFunction extends CFernTransformFunction

class CFernTransformFunction extends ((Double, Double) => (Double, Double)) {

def rnd = Random.nextInt(100)

def transformPoint(p: (Double, Double), a: Double, b: Double, c: Double, d: Double, s: Double): (Double, Double) =
 ((a * p._1) + (b * p._2), ((c * p._1) + (d * p._2) + s))

def apply(x: Double, y: Double) = {
 rnd match {
 case n if n <= 1 => transformPoint((x, y), 0.0, 0.0, 0.0, 0.16, 0.0)
 case n if n <= 7 => transformPoint((x, y), 0.2, -0.26, 0.23, 0.22, 1.6)
 case n if n <= 14 => transformPoint((x, y), -0.15, 0.28, 0.26, 0.24, 0.44)
 case n if n <= 100 => transformPoint((x, y), 0.85, 0.04, -0.04, 0.85, 1.6)
 }
 }
 }

def top = new FernFractalFrame(TransformFunction, 400, 400, 10000)
}

To run this example:

$> scalac FernFractal.scala
$> scala -cp . FernFractal

Go Fern Fractal

I’m so stuck in system level programming in C and Go I almost forget how to do ‘normal’ stuff ;-P Therefore  I gave myself a small Go programming exercise, something with visual feedback. Recently I read this excellent post on how to draw a fractal in Clojure and I thought it would be fun to rewrite this in Go.

The fractal is a Barnsley Fern and is an example of an iterated function system. You take a  function which returns some values and you use the returned values as the input to the same function in the next step, and you keep doing that until eternity or you put a limit on the amount of steps.

With the Barnsley Fern fractal you have four transform function with input x and y. We give it a start point x and y, one of the randomly chosen transform functions will calculate a new x and y. We draw this as a point on an image and than give the new x and y the the next randomly chosen transform function and repeat this 10,000 times.

If you do this with specific parameters you will draw a fern:

And here is the source code:

package main

import (
 "fmt"
 "image"
 "image/draw"
 "image/png"
 "image/color"
 "log"
 "os"
 "math/rand"
)

func main() {
 m := image.NewRGBA(image.Rect(0, 0, 400, 400))
 blue := color.RGBA{0, 0, 0, 255}
 draw.Draw(m, m.Bounds(), &image.Uniform{blue}, image.ZP, draw.Src)
 drawFern(m, 400, 400, 10000)
 f, err := os.OpenFile("fern.png", os.O_CREATE | os.O_WRONLY, 0666)
 if(err != nil) {
 log.Fatal(err)
 }
 if err = png.Encode(f, m); err != nil {
 log.Fatal(err)
 }
 fmt.Println("Done")
}

func drawFern(m *image.RGBA, x float32, y float32, steps int) {
 if steps != 0 {
 x, y = transform(x, y)
 drawPoint(m, x, y)
 drawFern(m, x, y, steps - 1)
 }
}

func drawPoint(m *image.RGBA, x float32, y float32) {
 b := m.Bounds()
 height := float32(b.Max.Y)
 width := float32(b.Max.X )
 scale := float32(height / 11)
 y = (height - 25) - (scale * y)
 x = (width / 2) + (scale * x)
 m.Set(int(x), int(y), color.RGBA{0, 255, 0, 255})
}

func transform(x float32, y float32) (float32, float32) {
 rnd := rand.Intn(101)
 switch {
 case rnd == 1: x, y = transformPoint(x,y, 0.0, 0.0, 0.0, 0.16, 0.0)
 case rnd <= 7: x, y = transformPoint(x,y, 0.2, -0.26, 0.23, 0.22, 0.0)
 case rnd <= 14: x, y = transformPoint(x,y, -0.15, 0.28, 0.26, 0.24, 0.44)
 case rnd <= 100: x, y = transformPoint(x,y, 0.85, 0.04, -0.04, 0.85, 1.6)
 }
 return x, y
}

func transformPoint(x, y, a, b, c ,d , s float32) (float32, float32) {
 return ((a * x) + (b * y)), ((c * x) + (d * y) + s)
}

Moving targets

I’ve been working on SCTP (Stream Control Transmission Protocol) for a while now. Much error and learning involved. One of the things I discovered (or better said; learned the hard way) is that some functions needed by SCTP are deprecated in the last 12 months. My knowledge was based on the excellent Unix Networking Programming book (also known as UNP). And sctp.h in FreeBSD 8.2. However in December 2011 RFC 6458 got published. This document describes the mapping of SCTP into a sockets API.

The functions I used most: sctp_sendmsg() and sctp_recvmsg() are now replaced by sctp_sendv() and sctp_recvv(). (As of FreeBSD 8.3/9.0) But more interestingly the default way to send and receive messages can now be done with recvmsg() and sendmsg(). These two functions, according to UNP (14.5 p390), are the most general of all the I/O functions.

A quick look reveals these methods are in existence for a very long time.

So I’ve started reading RFC 6458, upgraded to FreeBSD 9 and am getting familiar with the sendmsg() and recvmsg() functions.

Olivier