In the sea of projects that network engineers can work on in an organization, one of my favorites is a site build or a move. I am always interested in designing and implementing something new, especially when it ties into a larger ecosystem. Originally, this blog entry was going to dive into specifics of a data center move, but I realized that there five important areas that apply to network moves and builds.
As a network magician, you are expected to pull a design out of a hat, wave and turn it into brick and mortar. Ok, perhaps it’s not going to be that easy, but I am hoping to point you into the right direction.
You might be thinking “Well David, having a proper design is pretty obvious.” You would think right? Even though I can only control what I am responsible for, having a vision of the plan and even pieces that belong to other teams will help you in your implementation. “But David, isn’t that the job of the project manager?” To some extent. Also, if you have a project manager, congrats! Sometimes you are the project manager. I’ve been lucky to work with knowledgeable IT project managers as well as being the one to coordinate and implement.
Proper planning is not just your responsibility. If there are other teams that have work to be done during or after the move, their plans or lack of planning can affect you. Sure, those other areas are not in your realm, but there might be something the server team didn’t think of. There might be something the support techs didn’t think of. If you thought of those things that they did not, ask the questions. Have you reached out to other teams to see how things are going?
In some of my builds I have to rely on the server team to deploy their standards. If they fall behind, I’ll fall behind when it comes to some of the configuration. Reaching out to other teams and checking in doesn’t hurt and it always helps you build rapport with others. Working in silos will only add delays. There can never be too much communication in this kind of project.
For the data center move, having server lifts made life so much easier. If we had arrived at the new data center and all the server lifts were in use by others, what was option two going to be? No one wants to lift servers up to the top of the rack. Asking those questions and making sure a server lift would be available on move day was almost forgotten. It might not have been a big deal until it was needed. Then someone will always say “Why wasn’t this planned for?” Communicate and when you are done, communicate some more. This will ensure your plan takes shape.
The Early Config
I’m ordering a Catalyst 9500 for my new site. I can send it to the new location, but the network closet will not have a door for a couple of days. Ok, minor inconvenience. I’ll toss it on the rack anyways. Sure, it will be a little dusty because of the construction, but that shouldn’t hurt. Finally, I have a few spare hours to configure it the night before go-live – but it looks like we forgot to have the electrician run the right type of power.
All the above would be my deal breakers in sending equipment to a new site. Again, some of these things might seem obvious, but in the middle of orchestrating this project, those obvious things can fall through the cracks. All these feed into my first tip of timing. Ensuring your new equipment arrives to a safe location is important. Having electricity is not an overrated concept. It’s essential.
Prior to the data center move last year, we had all the equipment arrive at the office. The data center cage/racks were not ready and having access to the equipment just made staging and configurations easier to manage. I did not wait for go-live to configure the equipment; it was completed weeks in advance. I even simulated the environment within GNS3 just to make sure I had a full picture of what things would look like when they were tossed on the rack and powered on. Whenever possible, configure early. Yes, there will be those odd-ball projects that will not give you those opportunities. You will end up configuring equipment hours before a go-live or during some 2-hour window. Whenever possible, configure early.
I have a bad habit of telling the family how long I think an implementation is going to take. “I should be back by 14:00.” 14:00 comes and goes. I’m still not home. Well, sometimes Murphy’s Law takes effect. Anything that can go wrong, will go wrong.
The key here is to be prepared. Just like a sports team will practice and go over plans for game night, you should do the same with your go-lives. The day or night of a go-live or site build, you must be ready for anything. Do you have spare ethernet? Do you have spare fiber? Sure, those fiber connectors are expensive, but having a spare means the director doesn’t have to make an hour and a half drive to purchase one from the only place that carries them. I’ve been there. I’ve seen it happen.
You should already have the phone numbers for technical support ready to go along with serial numbers for devices. I’ve seen brand new equipment arrive at a site and be dead. Sometimes we want to be superheroes. I love troubleshooting and solving problems, but when you are running out of time in a change window, it might be best to seek external help. This is why IT pays a lot of money for support. During a move, a pair of firewalls decided that each High-Availability member wanted to be the primary. I am not sure why, perhaps a bug. At the same time, I was dealing with other issues and decisions, so I decided to give support a call and let them figure it out. The more eyes and hands, the better. After some rebuilding of the HA config, all fell into place. Sure, I could have rebuilt HA, but it was not something that was readily in my thoughts at that time. Be prepared for anything.
Cleanup on Rack 2
You’ve completed your change with time to spare. An environment has been built or shifted over to a new location. The end…or is it? There is still much to do after an implementation. It might or might not fall on your team’s shoulders, but cleanup and testing need to occur. Yes, literal cleaning of the space you are working in. Don’t leave a mess. With a new site, things should hopefully be organized, clean enough to appear in a magazine. You might have outsourced cabling to a third party. Usually those pros do a wonderful job in labeling cables and making everything look nice and tidy. If this wasn’t one of the outsourced tasks, it might be up to you to ensure cables are labeled.
Sure, you configured port descriptions (hopefully), but labeling cables is also part of the implementation experience. Take pictures of your implementation, especially if the site is remote. This will help you in the next tip of documentation. You do not have to be a professional photographer, but some cell phone pictures are better than nothing. How’s your bandwidth looking? You might have already tested bandwidth during the implementation of the new circuit, but now that there is some use to that circuit, is everything working as expected? Other teams might already be rushing to sign off on the project, but you might want to take some time to test different services with them. They’ll either call you the annoying engineer or the responsible engineer, but it is better to run into these issues before you hand the keys over.
The capstone to any successful (or perhaps unsuccessful) project is the documentation. Just reading about the need for documentation is probably making your hair stand. Don’t browse away just yet; hear me out.
Years ago, when I first started driving, I did not have a GPS in my car. My cell phone’s only capabilities were to make calls and send texts (shocking to some of you). However, I had met someone who today is my wife. During our first date I was to meet her out in the suburbs about an hour away. I hopped online and printed out a map. It pointed me to the interstate, which I had never driven on. There I was doing my best to follow the speed limit and juggle printed out instructions of my destination. I was grateful for those papers because it showed me the way. This is what proper network documentation does, shows you the way.
Ok, so all you are doing is a site move. Is the equipment the same? Did the IP addresses change? An extra uplink? Even if it is all the same, the address is probably different. Is that noted somewhere? Make sure to take the time to update or create documentation. Some people dislike this part of networking, but it is of utmost importance. Even if you have everything memorized, think of those who will come behind you. Putting together proper Layer 2 diagrams, routing diagrams, rack diagrams, wireless heatmaps, and even emergency contact info is essential. Does it take time? Yes. Make sure the PM has it as a deliverable. If you are the PM even better; don’t close out the project without completing all necessary documents. Someday you or another team member will need something to show them the way and they will be thankful to have it.
In the sea of projects an engineer might be swimming in, site builds or moves require some extra attention. There are many moving targets. There can be many teams involved. It’s almost like putting together furniture from your favorite complicated furniture store. By meeting and communicating with your team and other teams involved, you can be sure you have a reliable plan.
During the implementations, be prepared for anything. Having spares and help close by can pull you out of a jam. When you think you are done, you really are not. There are still cleanup items and baselines to be taken.
Finally, document everything. In my opinion, your project is not complete if the documentation has not been finalized. These five tips that I have briefly written about are columns that sit on top of a foundation. What is that foundation? Timing. The foundation to big projects like site moves or builds is timing. On my next blog, I will write about how timing is the most important tip that will make or break your site build.
To learn more about Networking Training, visit https://go.skyline-ats.com/ciscotraining