You are “dev” to me

Introduction

The idea of keeping a production and development network always wears on me. Unless the change management  is in place to offer some assurances that both threads are kept exactly in sync, they inevitably turn into a game of “not good enough to buy down risk.”

That was the rationale behind creating a system that allowed rapid captures of running machines, transitioning to a closed network, and with no changes to the network/systems have them be as they were just minutes earlier. Thanks to technology from EMC, Cisco, VMware, and VyOS this is all very possible.

Enter “Test Network”

The number of VLANs and separate network complicates things. The goto configuration would be to simply generate a new VLAN and dump the machines into it, but what happens when the machine you copying over expects routing in place?

The entire point of test is to reduce the risk that your changes will have adverse effects. Your test configuration should always have sufficient fidelity to mitigate the level of risk you and leadership are willing to accept.

Yippie for software routers! Yeah for Linux!

Vyatta unfortunately got gobbled up by Brocade a few years ago. Luckily a very talented team continued forward with the last open-source version and have marched forward VyOS.

Using VyOS to mimic an expansive Cisco vSwitches/Routing architecture allows you to not worry about the rest. A single VLAN can be spun-up to keep it “physically” separate while attaching sufficient interfaces to the VyOS box to make it the new core.

The process then becomes (either HITL or scripted) the below:

  1. Identify the systems to test
  2. Take latest VMware Snapshot or EMC Avamar in this case and deploy BMR on machine attached to VLAN of Test Network
  3. Deploy peripheral machines for support
  4. Start!

Simple Diagram Overview

 

Blank Flowchart - New Page

Lessons Learned

Some things to keep in mind if you decide to move forward with a similar configuration:

  • Be aware of software licensing before moving systems. Ensure the vendor provides support for testing with the license you now run.
  • Servers are often supported by peripheral services such as AV, AD, and others. Be sure for a clean environment you include those in your scripts for copying over.
  • VMRC helps a ton in Vsphere 5.5 and higher, but in lieu of this ability you will need some means to transfer files back and forth.
  • 10 NIC is the most you can attach to a VM. This will complicate things with VyOS. Keep in mind you only need the 10 networks you are using and not ALL of them at once. So A, C, E, F instead of A-F.
  • This can be replicated a thousand times over with only your capacity being the hinderance. Test Network 2-7 are okay too. Just replicate the configuration.
  • Active Directory Tombstones will bite if you are not careful. 60 days and you should plan on dumping and starting fresh.
  • Nothing stipulates you cannot migrate a Test Network to Prod. Keep in mind that in this method the copy goes both ways.
  • Systems with external interfaces will need simulators to mimic those connections. Especially important in manufacturing scenarios.
  • System load is usually a big impact point to keep in mind when testing configurations. Be sure to include ways to force system load to match enterprise. Maybe I will write a blog post on the various SQL, HTTP, SSH, etc. ways to do this with some python code..

Benefits

  • You are guaranteed to have a test network that represents your production network.
  • The on-demand nature ensures that storage and CPU cycles are only used when needed.
  • Depending on the size of the machine this can take any where from 30 minutes to 2 hours, but it is quicker than spinning a new environment.
  • Serves as a great disaster recovery test.

blgSixb

Liquorice Stick and Memories

“Mom, I don’t want to play clarinet! It is for girls!” I remember yelling at my Mother as I prepped for band my middle school years. We were poor growing up and the only option I had to afford an instrument and be in band was to use the hand-me down clarinet. My Grandmother had spent a decent amount of money when my Mother was the same age as I was then on a Buffet Crampon. For whatever reason I had associated some sexuality to all wind instruments and as such considered this suggestion from Mom an attack on my very blossoming manhood.

I never truly embraced the instrument until a visit to Tennessee. I suppose my Mother had spoken to my Grandfather or maybe I had mentioned it, but he proceeded to ask me about my liquorice stick playing. “Liquorice stick?” I had never heard my female centric clarinet called that. “Oh yes, Benny Goodman played a liquorice stick like no ones business.” Grandpa Hardie then proceeded to play a few big band songs where my much despised clarinet was central to the song. I had never heard music like that and I immediately latched onto it. My Grandfather gave me a love for the music and an identity for this instrument I never would have found on my own. He taught me that day, and many others, that being a cool guy was more about owning the role than what others thought about it.

There are countless other memories of my Grandfather I could share. Westerns, Cheezits, growing up in the hills of Kentucky poor, etc. But I will always cherish knowing my Grandfather was the guy who instilled in me a sense of charm and wit. He had a certain demeanour that always oozed a sense of calm and untroubledness that I have attempted to replicate in my adult life.

I have struggled to put in words the feelings of losing him this week. One of our last lucid conversations was him telling me about his life being complete. “I have done everything I wanted and I am ready.” He knew more than I did his time was coming to a close. He never wanted some big thing on his passing. His wish was for everyone to be happy and joyous for the amazing life he lived. No man is without faults or shortcomings, but I hope I can replicate, in addition to his composed charm, being able to close that chapter of my life in the same dignified and polished manner.

I bought another “liquorice stick” just like the one I grew up with. A Buffet Crampon E11 made in Germany. His passing reminded me that we all need a little jazz and big band in our lives. Grandpa danced to many a Benny Goodman songs in his life with a giant smile on his face. When I play and when I think of him, that is the memory I want to be present in my mind for him.

I will miss you Grandpa.

home

Withings Home Camera Review

I am always in search of products that help me automate and secure my home. I have run a mix of Foscam, D-Link, and Panasonics cameras dumping to a Linux server for the longest time, but it was clunky and not without its horrendous maintenance. Additionally the cameras, as aptly pointed out by Wife, were less than aesthetically pleasing.

carroussel_home

Enter the Withings Home Camera setup. It checked off most of the boxes of requirements I had moving forward:

  • Doesn’t look like an ugly box with wires and lights when sitting on the shelf
  • Local streaming – I want to view the video via WiFi versus going to the internet to retrieve. I have a slow rural internet connection and video out is not an option
  • Ethernet – I don’t care for wi-fi if possible
  • Local storage – If I want to hook up a drive for local storage, I should be able to
  • Power Over Ethernet (POE) – This one unfortunately is getting very hard to find
  • Wide view lens – I hate having to mount multiple cameras just to cover a single room

Just for those that find this looking for reviews, it is a great setup. You can tell it is still a new product, but compared to the homegrown solutions you cannot go wrong. It is missing a few things such as IFTT, external API, Motion/Alarm times (if I am home I don’t need to know I am in the livingroom), and the ability to lock down ports for external access. The VOC air quality thing is nifty, but the propane heater in the house drives it batty. Two way comms for the audio is good when it works, but not very reliable on less than desirable WAN connections. WebRTC is the backend so I believe that is partly to blame as the comms channel matures.

From a security standpoint….

  •  I run a pretty tight firewall setup. I noticed external video access while away wasn’t working. A quick traffic dump showed the device was attempting to use ports 1024-65000 randomly. I emailed support and was told this is how it worked and to open all the ports. Nope. Fix it guys.

B75NCvACMAE00QF

  • XMPP is used heavily for the messaging and transfers which is nifty. Everything is encrypted via SSL on a secure XMPP port. You can see from the dumps that the timeline feature is all exchanged via XMPP to this server: prod-oregon-timeline-2-days.s3.amazonaws.com
  • Each device is given a unique ID for sending the messages and retrieval. I haven’t tried spoofing or decoding, but here is the XML stored on the server:
<Error>
<Code>AccessDenied</Code>
<Message>Access Denied</Message>
<RequestId>8F0D4A4C05EDE740</RequestId>
<HostId>
YCLYIVb8rwnRYWQHZjP6SCSHV6smaU0k7uSaW8N1zX8RKScWjgD3MycckIMzAIiuil89LK5xLn4=
</HostId>
</Error>
  • Remotely checks into http://scalews.withings.net/ for status. I assume this is firmware or upgrades, but have not validated
  • When there is an alarm it uses a cgi-bin against the same scalews server for storing the alarm

Example Alarm

 

AKR007_900

Smitten with the Mitten – 6 Months in Michigan

Moving here was the culmination of a vacation turned scouting trip. Having spent the past 4 years living abroad in various countries it was time to visit the Vaterland and bring my Wife to meet the family.

That was until I decided I wanted and we needed a change.

Having now spent 6 full months here I wanted to share the things I have learned since moving here. It is by no means an exhaustive list, but they are the things that have struck me as a new Michigander.

  • When you live in the breadbasket you get spoiled by a fresh bounty. I on the other hand have not prior so the food here has amazed me. Everything is fresh, local, and bursting with flavour. My Wife and I have traveled the State and some of the restaurants we have stumbled upon are world-class. Our biggest fear as foodies was that we would be bound to home cook the rest of our days, but that is simply not the case. The food here is excellent.
  • If you turn on one of our German or Dutch playlists and lose track of time, you could mistake your drive across the country for being there versus Michigan. The scenery can be jaw dropping beautiful. It is a relatively flat state, but the way the sky dances against the vastness of fields and forests adds a uniqueness to it all.
  • Much of my life has been spent in the South where the mantra always goes that they are the friendliest people in the US. The North gets a bad rap about being a cold and stoic place, but I think that is undeserved. The several times my Wife and I have left a conversation with a stranger thinking “gosh they were helpful or friendly” are countless.
  • Art is appreciated here. We enjoy the ballet and theatre and there are opportunities to see just about anything within an hours drive. Between Detroit, Midland, and Grand Rapids you options are open to doing something of that type of event. The productions are smaller and less grandiose, but they are well done and the community feels a sense of ownership due to that sizing I believe.
  • The State seems generally conservative, with a liberal leaning. It is hard to be a Union state and vote red. I would consider myself an independent libertarian and had thought I would be socially cast aside here. I am an oddity in the North part of the State, but overall people seem welcoming. In addition, people are just as fine to let you be. I can disappear onto our acreage and no one would think twice about involving themselves in our matters. I like that. Very different from the metropolitan.
  • Family being right up the road is such an amazing thing. This isn’t purely a Michigan thing, but it is something I love. With the months getting cold it seems the State binds together to have many indoor activities. There is always a cousin with some school show/sport/activity to go see. We walk into bars or restaurants and run into people we know. That sounds like Cheers, but in our case it is Cheers with the Schmidt’s.
Now things that I thought I’d love, but haven’t.

  • Sorry guys, but the beers and wines are overrated. As a snob in both arenas, I have yet to be impressed with the available refreshments. Trust me, I have tried EVERYTHING I can get my hands on. My basement fridge is littered with 5-packs after trying a beer from down the road only to be disenchanted. As a big Dutch, Belgian, and German drinker I had hoped the roots of the state would offer some good stuff. Been disappointed, but I will keep trying. I am never one to turn down something new and possibly tasty.
  • The roads…oh god the roads…I take delivery of my Ducati 1299 in a few months and I have started to scope routes, not based on their twisties or straightaways, but rather on if I can ride down them safely. I have lived in other cold places with lots of snow. Michigan doesn’t have some special brand that makes this difficult. For the love of everything beautiful in this place, FIX THE ROADS!
  • If you look at the most recent statistics, young Michiganders are leaving the State in droves. There are not enough opportunities to keep them here and as such the impending baby boomer retirements will mean a shortage in my opinion. We must attract and provide for young talent.
  • Embrace “going green” for the sake of your residents. Driving through the Thumb you see some of this through windmills and solar arrays, but Michigan (and the automotive industry) could stand-out if they truly took hold of this ideology. The great expanses between cities are begging for a series of charging stations or similar. Geothermal and wind power should be blanketing this state as the Dutch and Swedes have with their similar climates. As my Grandfather says “I am a farmer and going green is what I have done my whole life.” Just makes sense for this region.

Pure Michigan is a cheesy marketing campaign, but one I fully understand now. The State is an oddity of sand dunes, skiing resorts, metropolitan cities, and farming land. People remember it for the shape, but I think they are missing out on what is inside the mitten. We are looking forward to exploring more after the winter months thaw out. We can’t wait to see what else this beautiful state has in store.

This slideshow requires JavaScript.

Traffic Control on Linux with FireQOS

In order to make full use of my half-duplex WiMAX link, I started looking for anything and everything I could use to optimize it. Linux has some pretty decent utilities with iproute2 and netem to handle these type of configurations. They don’t compare to OpenBSD’s PF, but they work once you get the setup in place.

Due in part to how ugly TC is out of the box, I really like FireQOS for defining the basic configuration. The developer also makes a great iptables wrapper called FireHOL, but iptables rules are easy enough to write in my opinion.

I wanted to share my FireQOS script for those who may have similar needs. It prioritizes my traffic with the following basics:

  1. VOIP – Commits 30kbit no matter what else is going on. This is sufficient for a single G.729 SIP connection
  2. Interactive protocols such as DNS, SSH, and XMPP receive 20% of the bandwidth
  3. Facetime – Committed 200kbit for video streaming
  4. VPN connections (PPTP/IPSEC/OpenVPN) are given 20% of the bandwidth
  5. Surfing is given majority of bandwidth with 60%
  6. Synacks, Mail, and Usenet bring up the rear with whatever is left. They will consume available bandwidth if nothing else is going on

As I write this my network is function just spiffy with no issues streaming via SONOS and downloads via USENET. I am very happy with the config.

From /etc/firehol/fireqos.conf

DEVICE=enp3s0
INPUT_SPEED=1910kbit
OUTPUT_SPEED=200kbit
interface $DEVICE world-in input rate $INPUT_SPEED $LINKTYPE
 class voip commit 30kbit
 match udp port 5060 
 match udp dports 10000:10100
 match sports 3478,5349
class interactive commit 20%
 match udp port 53
 match tcp port 22
 match tcp port 2222
 match tcp port 8022
 match icmp
 match tcp sports 5222,5228
 match tcp sports 5223
class facetime commit 200kbit
 match udp ports 3478:3497,16384:16387,16393:16402
class vpns commit 20%
 match tcp port 1723
 match gre
 match dport 1195:1198
 match tcp port 1701
 match tcp port 500
 match udp port 500
 match udp port 4500
class surfing commit 60%
 match tcp port 80
 match tcp port 443
class synacks
 match tcp syn
 match tcp ack
class default
class mail
 match tcp port 143
 match tcp port 993
 match tcp port 25
 match tcp port 465
 match tcp port 587
class usenet
 match tcp port 563
interface $DEVICE world-out output rate $OUTPUT_SPEED $LINKTYPE
 class voip commit 30kbit
 match udp port 5060
 match udp sports 10000:10100
 match dports 3478,5349
class interactive commit 20%
 match udp port 53
 match tcp port 22
 match tcp port 2222
 match tcp port 8022
 match icmp
 match tcp dports 5222,5228
 match tcp dports 5223
class facetime commit 200kbit
 match udp ports 3478:3497,16384:16387,16393:16402
class vpns commit 20%
 match tcp port 1723
 match gre
 match sport 1195:1198
 match tcp port 1701
 match tcp port 500
 match udp port 500
 match udp port 4500
class surfing commit 6%
 match tcp port 80
 match tcp port 443
class synacks
 match tcp syn
 match tcp ack
class default
class mail
 match tcp port 143
 match tcp port 993
 match tcp port 25
 match tcp port 465
 match tcp port 587
class usenet
 match tcp port 563

 

Binary makes me giggle