Day 2 at The NDC London

This is an old post from December last year:

Below is my summary of the key take backs from each talk I attended for Day 2 (and my final day of the 3 day conference) at the NDC in London.

Continuous Delivery of Databases – Enrico Campidoglio

Really good talk from Enrico on this topic. Enrico started by taking us through some definitions on continuous delivery and deployment, and all the integration points in between on the pipeline. He then started to explain how database deployments are hard, and they are hard because generally, they are manually done by us humans, and humans are prone to mistakes. In quoting Enrico on that, in his Yoda style “If manual it is, run continuously, it cannot”.

He then started to explain why DB deploys are so hard due to the rigid relational model with concerns for data integrity, it simply wasn’t designed to be changed incrementally.

So the solution to this can be achieved in 4 steps; source control your schema and data, decide how you’ll make changes, track the version of your DB and deploy it all automatically.

Enrico then explained 2 approaches to achieving this, 1 was through the state model, which is a snapshot of your DB at a time, deployment happens by comparing the snapshot with the destination DB and creating a transform script. Tools exist such asRedGates SQL Compare to tackle this. But a better way, is using the transformation model. This works by taking a specific script order migration of a partition of the DB and versions and deploys the changes in that specific order. Tools used for this can beDBUp and Roundhouse.

If you use the transformation model, you should separate your DB migrations into a separate app to your app that uses the DB. This app can then run on its own separate contextual pipeline and deploy changes separately to the main app. If you use something like DBUp, it is a library that you import into your project to help with the migrations, Enrico then demoed a very easy way of using DBUp to deploy separate DBs and versions without impacting a web app.

Enrico finished by highlighting 4 key areas to keep in mind when tackling automated DB deploys, 1) Treat your DB as source code, 2) Prefer additive changes, 3) Deploy the DB independently and 4) Have a recovery strategy such as roll-back and roll-forward.

Deliberate Advice from an Accidental Career – Dan North

As always, from a talk by Dan North, this was a little difficult to summarise, but I’ll give it a try.

This was a real thinking and contemplative talk from Dan. Dan took us on a journey, a journey of what types of eureka moments allowed him to focus on how to restructure his thinking and help pay it forward to fellow professionals in our field of work. But it was more to do with people on his career journey, in what they said to him that made him think this way. He highlighted 9 points in his life:

  1. The Outage… “Today we learn”, here there was a DB outage caused by Dan in his early days, but his senior didn’t take a blame approach, but stayed calm and used the mistake as an opportunity to learn.
  2. The Referral… “I sent them your CV”, here was a time in which a friend of Dan’s went through the hiring process at ThoughtWorks, but failed and sent on Dan’s CV afterwards even though Dan felt he’d never go for a roll like that, he didn’t believe in himself but his friend did. Sometimes you are better than you give yourself credit for.
  3. The Coach… “Hey buddy!”, This was a time when a junior member of the time would continually ask questions about work and have someone pair with him to get it done, over time, the role would reverse and the junior would be asked to help the more senior person, this is because a level of empathy has been created between the 2, a perspective has been created.
  4. The Stack Trace… “Let me figure it out”, This was a time, Dan noticed a developer approach a pairing problem with another person and state this, when something failed by analysing the stack trace rather than listening to writer of the code explain why it happened, because a stack trace can be information and learning on a journey of the application.
  5. The Product… “Nice, is it live yet?”, This was a time when Dan wrote some code and wanted to tell the world, put was put down when asked the question, because we need to know when we are building something and how much value it is adding in terms of business value.
  6. The Leader… “You’ll have to teach me”, here was a time Dan was faced with feeling he wasn’t challenged enough at the company, he had all these great ideas, but nothing was happening. So it was time to move on… until he had a meeting with the CIO who wanted him to work on a new role in changing the team culture structure and get away from silo teams. But the CIO asked Dan, to teach him how. This was a moment that a realisation that even the most senior in a company have vulnerabilities and are willing to learn from more junior staff, and experimenting with new ways of doing things (similar to Doc Norton’s talk yesterday).
  7. The Interview… “Don’t Hire Yourself”, a lot of people fall into the category of hiring someone they feel connected with, someone with similarities instead of hiring someone who can add something different, something better.
  8. The Customer… “Let’s ask Phil!”, one time, there was a request put in to figure out how we deploy, what should we do, Dan and his team approached the ops guy, who, because he knew one way of doing it, that was the way it would be done. This can prove problematic when we silo important knowledge such as deployment to one person and down the line that person leaves the company.
  9. The Start… “How soon do you want feedback?” Ask the team to list onto two cards the scariest and most immediately valuable (can deliver as soon as possible) items they can think of and write them down, a lot of the time you will find commonalities on both cards. From this the team can realise what they can work on immediately, sometimes the answer to the question can be answered by the team, and this is similar to point 6.

From all these learning moments in Dan’s career, he then took them to summarise what we should take forward and how we can help others by, sharing knowledge as we are what we do, choose your travelling companions in your career, be open to opportunities and believe in yourself, teach what you have learned and pay it forward, be yourself and Practical triumphs inspirational. People will forget what you said, forget what you did, but never forget how you made them feel.

Polyglot Heaven – How to build an Async, Reactive Application in 2014 – Tomas Jansson

This was a really enlightening talk from Tomas on essentially an approach to being a polyglot on an Async CQRS architecture.

Tomas began this talk by asking the question, why be a polyglot? Well, when all you have is a hammer… Being a polyglot allows you to choose the right tool for the job. If you have documents, you use a document DB, if you have changes, you use an event store, if you have relations, you can use a graph DB.

With all these tools, we can take a divide and conquer approach at the architectural level using CQRS. You must look at your application and figure at how much of it is functional, and turn all that functional into functions, functions are easy to test and we can send our function commands as events to something like event store, which with its subscription based streaming allows us to pull that data off, notify the UI of a change if need be and store the data in ElasticSearch and Neo4j.

Tomas then took some time to demo to us a use case of applying these technologies together to illustrate how we can use them in a typical ordering system with customers, orders and their relations. We were shown some basic cypher based create methods for adding the new orders to Neo4J and also how to use cypher to query Neo4j. Tomas then demoed how easy it can be to create a recommendations engine using Neo4j, considering relations is its thing, this is something that wasn’t too difficult to show. He also spoke about using ElasticSearch for applications requiring auto-complete, as it is lightning fast and is purpose built for querying.

After the demo, Tomas posed the question again, why polyglot? To which the answer is simple, not all data is the same, and everything isn’t a nail.

Tomas finished with providing us with a link to the demo, so we can look at how to build an architecture using EventStore, Neo4j, ElasticSearch, C# and F#, which can be found at:

Agile Patterns: Estimation – Stephen Forte

Stephen presented this really interesting talk wearing a rugby jersey, he explained that he always wears a rugby jersey when speaking about Agile.

He began by taking us the problems with estimations on unbreakable schedules where deviation is considered bad and how traditionally too much time is spent on defining a specification which is not complete. He then reminded us of the cone of uncertainty, which has been around for some time now, but still plays a big part in starting to get estimation right. We should be focusing on re-estimating a project after each iteration. But how do we actually estimate?

There a few components involved in estimation, 1) User stories and having the PO involved as your business proxy. 2) Story points – we are better when estimating with objects rather than numbers, like using t-shirt sizes, so long as the team understand that for example, a small t-shirt size is equivalent to something trivial like a login screen, or 1 story point is a login screen, so one piece of functionality could take 4 login screens to complete. 3) Planning Poker, resist the urge to average everyone’s score, humans are prone to avoid conflict, re-vote when you can. 4) Product backlog, a story should only be a one-liner piece of functionality, start a project with a small number of estimated stories, as the more stories there are, the more likely to have stale stories. 5) Velocity, re-estimation and OBTWs (Oh by the Ways), after 4-5 sprints you have a better idea of team dynamics, meetings etc. to estimate. Something to keep in mind, is that just because Team A works to 20 points in a sprint and Team B works to 10 points, it doesn’t necessarily mean Team A is moving slower.

Stephen then presented a formula he uses to help re-tune estimations for the business and working to deadlines, so the business can decide whether or not to add or remove items from the backlog as the project progresses. The formula is pretty straightforward; by taking the backlog estimation minus the completed points in the sprint, plus the item points added to the sprint, these are the OBTWs, plus the bug points added to the sprint, minus the number of items removed from the product backlog, will equal the sprints to completion. You should also use the cone of uncertainty with the results to calculate the weighted sprints to completion.

Stephen finished by talking about how to sell this technique to business and explaining that, repeating this technique for 4-5 sprints as your backlog goes down will give you a better estimate of how many points your team can commit to and the number of sprints it will take, this will give the business a chance ahead of time to remove any stories they don’t need to meet a deadline.

Mobile web – Merging Responsive and Adaptive Techniques – Francisco Manuel da Silva Ferreira

I’ve always been a big fan of Bootstrap for building the responsive web, however Francisco really opened my eyes to new ways of building this responsive web and not relying on a framework in doing so! Bootstrap can be sometimes mis-used and mess up the html very easily. Instead of using a framework, use a CSS pre-processor likeLESS.

Francisco talked about he and the team at EBAY approach the responsive web with a little mix of the adaptive approach with their menu system. He explained how they use the concept of widgets for everything with the golden rule of a widget having a width of 100%. You can use LESS to create child widgets inside the parent widget, specifying a set width for the child elements or combine the child widgets with media queries to change their width based on a screen size. The idea of using widgets is such that, we can simply add, replace or move a widget anywhere on the screen and it will just fit in to the parent widget if dropped on the parent widget and re-size based on the parent widgets child widget constraints. Nearly everything on the EBAY pages are made up of widgets.

Html should represent the content that it has, and what components you’re dealing with rather than logic on how it should be displayed.

Francisco provided us with a link to his blog at the end which goes into more detail on the subject and how to implement widgets in a web app:

The State of DevOps in the .NET Space – .NET Rocks Live!

I kind of treated myself to this talk, I’ve always been a big fan of .NET Rocks & I certainly didn’t want to pass up the opportunity to see them live! In fact, and I’m not sure if I should admit this, but I listen to these guys podcasts most mornings while working out in the gym before work.

Anyway, onto the talk itself. Carl and Richard had 4 guests on this talk, Enrico from an earlier talk I had attended, Ben Hall, Peter Mounds and Jeff French. The panel first started by discussing the use of Docker and how Docker allows you to isolate functionality in a container, which can act as a sandbox in a lightweight VM and behave as if it works on its own OS inside an actual OS, and then you can just throw it away when needed.

Peter then spoke about dependency problems and the reliance on AD (Active Directory) in the cloud being something that can hold you back on things like SQL Server Replication, with an alternative being SQL Server Mirroring to get around the AD problem. AD is like having a pet, once you have it, it will be around for a while, AD is ok in your company, but when you move to production in the cloud, it’s best to not have a dependency on it.

The panel then started to speak about keeping things small, with solutions likemicroservices, and being able to simply throw a part away if it ends up being not fit for purpose. Enrico also highlighted in terms of his earlier talk about the data in databases, the data part making it hard to simply difficult to throw it away if it doesn’t work out. It’s difficult to let go of the relational model, but it doesn’t mean we shouldn’t have to try, apply the right database tool for the right job.

Next the panel went into talking how the dev teams should own the deployment and delivery process and be in charge of the application once it is in production, which means monitoring. Using tools like statsd and graphite can help greatly like this. Statsd is something we have introduced as well at bookatable, so it’s interesting to hear it doing good things elsewhere. Monitoring is possibly one of the most important parts of the DevOps culture.

At the end of the talk, Richard revealed a hidden gem in Visual Studio, which none of the audience knew about, called runtime analytics, which provides real-time performance data right down to the method level of your code and claims to not impact performance and can be configured to call an API with the data. From here we can use the likes of logstash with ElasticSearch and Kibana to track and store the data (as I previously mentioned in a talk from Day 1, see previous blog post)

Embracing HTTP in an era of API’s – Hadi Hariri                         

Another excellent talk to end day 2 at the NDC London. In this talk Hadi took us through what is and isn’t a ReSTful API and some of the more common misconceptions along with what we can do to ensure our API does indeed fall into the category of being a ReSTful API, which essentially means following the Richardson Maturity Model. ReST has become a bit of a buzzword and most people state their HTTP API is in fact ReSTful when it’s clearly not.

With APIs we should have only one endpoint, or one interface for our resources and our clients to talk to these resources. We need to embrace a move to resource based URLs to help enable this. We should focus on what our allowed verbs should be. Not every resource should allow every method such as GET and DELETE. When requests come in for OPTIONS in the request header, we should intercept that request and append headers to indicate what methods are allowed, or even, when a request comes in for a non-allowed method, we should respond with headers stating what is allowed.

We need to specify a location header for POST requests, to show the new resource location after the creation of the resource. When responding to the client with an error we should indicate the correct response error in the returned status code, such as unauthorised or internal server error to guide the user.

Content-negotiation should be built in, to allow the client to request what kind of response they want such as xml or json and even state the level of priority preference in the request header to indicate preference of xml over json or vice versa.

Caching should be built in – specifying max-age, expires and conditional GETs (Etags) – which is the hashing of the requested resource, then when a request comes in for the resource, a check is made to see if it has been modified, if modified, it returns a new resource if hashes between requests do not match. In case of caching, return a 304 not modified in the response header if the resource has not been changed.

HATEOS – Hypermedia as the engine of app state. We should be adding information about what happened to a booking for example and passing back the possible state changes the client can make along with the location of that resource, that way we are taking away the issue of the server handling state and allowing the client to worry about state. Hypermedia it the most important part of getting ReSTful right according to the Richardson Maturity Model.

We should be considering the media types we can support, such as custom formats, or existing formats like ATOM, or JSON with HAL and JSON-DL.

We should be supporting versioning, either through URL versioning, custom headers and accept headers. Always use SSL, along with OAUTH.

If we don’t have the points covered from above you can’t be truly ReSTful, you can have a HTTP API, but you won’t be ReSTful.


Thanks for reading and keep building awesome.

Until next time!

Ian B

P.S. You can also find me on Twitter: @ianaldo21

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s