NOC No More

One of the challenges with taking an organization in a new direction is the old guard. That was one of the challenges that I faced at D&B and it is a challenge that I face now.

As I was getting ready to leave work today I was drawn into a conversation about monitoring and alerting in this brave new world. Personally, I am against monolithic Network Operations Centers. In my experience, they prolong just as more problems than they actually help solve.

At D&B, we had two different ways to manage incidents since we had two different hosting models: the stuff that I managed and everything else. For the stuff that I managed, my team of system engineers were in an on-call rotation. Any alert that was triggered by our monitoring went (via PagerDuty) to the on-call admin who would diagnose the issue, call in any additional support such as a DBA (as needed), and resolve the issue. I would wager that 99.9 percent of the time, the on-call person could resolve it within moments without customer impact (this is born out by the nearly 1000 days of uptime I had when I left).

Everything else went through the NOC managed by our hosting partner. Once an app was alerted on, they would get call out to every possible group associated with an app such as systems, DBA, network, application team, service manager, etc, etc. I’d like to say that it took longer to get people on the call, but truthfully it usually took an hour or two because nobody knew enough about overall architecture (which is a whole different problem).

At Blackbaud, I’m hoping to move to take what I had in place at D&B to the next level. With our move to the cloud, it has given us the ability to be able to switch to cattle rather than pets. Long term, I hope to be able to eliminate alerting (in the traditional sense) and instead just trigger an action that terminates the offending server and just spins up a new one. Sure, we get notified that something happened and maybe even in some instances quarantine the offending node, but we get production back up virtually instantaneously.

Whatever the end state is, it cannot rely on people needing to interact with it to get it working again. If we can build them through automation and deploy them through automation, we need to be able to fix them through automation. At the end of the day, automation will fix them faster than even my old team at D&B could.


Being Accountable

One of the things that I plan to do this year with my blog is use it to chronicle my progress towards my goals as well as to hold myself accountable for the poor choices that I make.  I have had some time over the last few weeks to better clarify my goals for 2016.  My revised goals, as currently stated, are:

  • Weigh less than 200 pounds
  • Get four AWS certifications
  • Be able to jam with my son on a guitar
  • Complete the Austin Distance Challenge
  • Write.  Everyday.

Right now, my current thought is to post a weekly progress report on each of the five goals.  My progress so far has been pretty mixed.  I have started running again, although I have lost a lot of what I had before October of last year.  It’s funny how that happens.  My weight hasn’t changed much in the last few weeks.  I haven’t made any movement of the AWS certifications nor have I written everyday (although I have generally logged some comments in my daily diary everyday).  The only think that I have been pretty consistent with is learning to play guitar.

This week, I plan to take a few different steps towards my goals.  I will practice my guitar for 1 hour a day, I am going to run everyday, I am going to cut carbs out of my diet as much as possible, I will read three AWS white papers, and I am going to write every day.  I’ll also be working on a better way to display this in a more weekly format.


Goals for the New Year

One of the biggest things that I learned last year is the power of sharing your goals with other people.  There were a couple of things that I set out to accomplish in 2015 and I was much more successful reaching my goals one the ones that I shared with other people.  This year, I going to be sharing my main goals out loud, starting with this blog.


Read More Books.  For the last 18 months, the majority of my reading has been revolved around my school work.  Having finished my degree, I hope to do more recreational reading this year.  My goal for 2016 is to read or listen (I’m a big Audible fan) to 52 books.  This one is pretty easy to measure.  


Be more mindful of my eating.  I could probably write exclusively on the dysfunctional eating habits that I developed when I was a kid and have enough material to post every day for the rest of my life.  Rather than focusing on the past, however, I am trying to make a better future.. The year I am going to focus on being more mindful of what I eat.  This will be my biggest challenge. The will be measured by my weight loss. 


Run the Austin Distance Challenge.  When I was in the Army, I hated running.  At least the organized, everybody in a formation, lets run and sing for miles and miles kind.  Over the last few years I have learned to love running, especially the solitary nature of what I do now.  Often it is the only time I get to myself and my thoughts.  I have had a couple of false starts trying to run a half marathon, but no more.  I AM going to run the Austin distance challenge this year.  Progress will be measured by whether or not I actually compete.


AWS Certifications.  Having completed my MBA, my educational focus will be on mastering Amazon Web Services (AWS), with the certifications being the way that I will measure progress.


I’m excited about what 2016 has in store for me and my family.  It’s a time of crazy transition but I wouldn’t have it any other way.



Lots of Change

For the last few months things have been in serious flux.  November saw a pretty big change in my home life as my 4 and 5 year old grand children came to live with us while their parents try to get their lives in order.  It is a pretty serious change for us as our youngest is 17 and a junior in high school, so I haven’t had little ones running around underfoot in a lot of years.  It has taken a lot of getting used to.

This was followed by a change in jobs at the beginning of December as I took a new job as the Manager of DevOps at Blackbaud.  There were a lot of things that went into my decision to leave Dun & Bradstreet, but one of the main attractions for Blackbaud was the ability for building out a new team and helping define what DevOps means for the teams I support.

2016 is going to definitely be a different type of year.


New Strategies

For the last two years, working at Dun & Bradstreet has been a bit of a challenge.  In May of 2013 our then CEO Sara Matthews announced that she would be retiring as CEO and that the search would begin for her replacement. From an IT and strategic standpoint, that pretty much stopped everything that was going on.  Rightfully so, I guess, because who wants to start implementing a strategy only to find out in a few months that the new CEO has something totally different in mind.  In September of the same year it was announced that Robert (Bob) Carrigan would be taking the reigns of the company.

Of course, Bob needed time to get up to speed on everything, figuring out what needed to be done and how best to accomplish it.  In early 2014 Bob announced our new strategy and things seemed to finally be lining up for us in technology.  Shortly after the announcement was made, however, we received word that our CIO had announced that he would be retiring and leaving the company.  This wasn’t much of a shock, to be sure, but it still meant that we were back to not doing much of anything lest it all be undone once the new leader was on board.

Following about the same pattern, the announced Curtis Brown as our new CIO in September of 2014, and he spend the next six months shuffling people and putting together a new strategy for the technology organization.  Now that all the people are in place and the strategy is ready to go, I am really excited about the direction we are heading down.  I believe that we finally have the right leadership in place to make this into a truly world class technology organization.  The kind that sets standards not just for our industry, but for all technology organizations.

I’m really looking forward to it.


Configuration Management Tools

The time has come for us to re-evaluate our configuration management tools.  For the last 5 years we have been a CFEngine shop, and while we have looked at some of the tools that are available over the last few years, we have consistently come back to CFEngine.  The recent acquisition of companies by the mother ship has brought in a whole slew of management tools.  Over the past year, we have purchased companies or products that utilize Chef, Puppet, Ansible, and Saltstack.  

Over the next few weeks, we will be evaluating each of these tools in our environment.  My goal is to standardize on a common toolset for the enterprise.  My goal is to one day be able to push a lot of the configuration management down to the application teams and only focus more on setting standards for common components.  With all of the new requirements we are getting to support AWS, we need to ensure that the tool that we are using will support all of the things that we need for it.

This ought to be fun.


Thinking Out Loud

I’ve been thinking a lot lately about my blogging (or lack there of) and trying to figure out what exactly it is that is keeping me from getting beyond one or two posts a month.  After some thoughtful consideration, It comes down to a singular problem that manifests in two unique ways.

The singular problem is my internal filter.  The first way that the problem manifests itself is though the perfection angle.  I don’t ever feel what I am writing is good enough, so I self edit and eventually give up.  Lately, it has become clear to me that I am my own worst critic of my writing and that usually what I think is bad is pretty ok.  I’ve learned this through the course of working towards my MBA.  For the last few terms, I think I have turned in some sub-standard papers.  Work that I would have given a B or a C has consistently earned my an A.  I don’t think I am that brilliant and generally just chalk it up to my classmates being that bad, but I am starting to feel like I can be overly critical of myself.

The second way that my problem manifests is that I silence myself because of my job.  Part of me doesn’t want to reveal what is going on in so as not to irritate the power that be or reveal something that earns me the “OH MY GOD MARK WHAT DO YOU THINK YOUR ARE DOING!” call from my boss.  Not that I am sure many people are reading what I have to say anyways.  I suppose I am worried about revealing something that is contradictory to what a potential customer (or customer) was led to believe about our technological capabilities.  Like any organization, we are doing some really cool stuff and we have some stuff that is pretty broken.  We have issues with hardware, processes, and people, but we also have some pretty advanced infrastructure and robust practices.

I have to get better at turning off my internal filter and get better about sharing the lessons that we are learning, both good and bad.  I know that it is going to take practice and it is likely that some of my posts are going to be subpar as I force myself to go beyond my filter and just get content out there.  I’m confident that we will get better over time.


The Lifebuilder’s Creed

I recently came across The Lifebuilder’s Creed by Dale Witherington while watching a the introduction to John Maxwell’s Today Matters leadership training course, and it really touched my soul and I wanted to share.


The Lifebuilder’s Creed

Today is the most important day of my life.
Yesterday with its successes and victories, struggles and failures is gone forever.
The past is the past.
I cannot relive it.  I cannot go back and change it.
But I will learn from it and improve my TODAY.
TODAY.  This moment. NOW.
It is God’s gift to me and it is all that I have.
Tomorrow with all its joys and sorrows, triumphs and troubles isn’t here yet.
Indeed, tomorrow may never come.
Therefore, I will not worry about tomorrow.
Today is what God has entrusted to me.
It is all that I have.  I will do my best in it.
I will demonstrate the best of me in it —
my character, giftedness, and abilities —
to my family and friends, clients and associates.
I will identify those things that are most important to do TODAY,
and those things I will do until they are done.
And when the day is done
I will look back with satisfaction at that 
which I have accomplished.
Then, and only then, will I plan my tomorrow,
Looking to improve upon Today, with God’s help.
Then I shall go to sleep in peace … content.

It Could Have Been Worse

Things have been pretty trying for me the last two weeks.  Last Monday, just before leaving for work, my outside electrical panel had and issue and caused a fire in the panel as well as in my kitchen behind my stove.  The damage was relatively minor, thanks to the the fact that when it happened at 5am I was awake and hadn’t left for work yet.  Had either of those two variables changed, the odds are I wouldn’t have a house right now. I am extremely thankful to God that it was as minor as it was.

As these things go, it has been relatively smooth.  Our electrician had everything repaired/replaced the day after the fire and by the time he left we had power once again.  Our insurance company (Travelers) was out on Monday afternoon to start the claims processes and by Thursday we had our claim documents and we had the check a week after the fire.  The cleaners have been in our house all week cleaning up the smoke damage and the contractor is coming out tomorrow to estimate the kitchen repairs.  The stove was a total loss, and I cannot replace it until the kitchen work is complete.

Here is some of the damage:

IMG 0282 19 50 35 IMG 0283