You are dev to me

3 minute read

Introduction

The idea of keeping a production and development network always wears on me. Unless the change management  is in place to offer some assurances that both threads are kept exactly in sync, they inevitably turn into a game of not good enough to buy down risk.

That was the rationale behind creating a system that allowed rapid captures of running machines, transitioning to a closed network, and with no changes to the network/systems have them be as they were just minutes earlier. Thanks to technology from EMC, Cisco, VMware, and VyOS this is all very possible.

Enter Test Network

The number of VLANs and separate network complicates things. The goto configuration would be to simply generate a new VLAN and dump the machines into it, but what happens when the machine you copying over expects routing in place?

The entire point of test is to reduce the risk that your changes will have adverse effects. Your test configuration should always have sufficient fidelity to mitigate the level of risk you and leadership are willing to accept.

Yippie for software routers! Yeah for Linux!

Vyatta unfortunately got gobbled up by Brocade a few years ago. Luckily a very talented team continued forward with the last open-source version and have marched forward VyOS.

Using VyOS to mimic an expansive Cisco vSwitches/Routing architecture allows you to not worry about the rest. A single VLAN can be spun-up to keep it physically separate while attaching sufficient interfaces to the VyOS box to make it the new core.

The process then becomes (either HITL or scripted) the below:

  1. Identify the systems to test
  2. Take latest VMware Snapshot or EMC Avamar in this case and deploy BMR on machine attached to VLAN of Test Network
  3. Deploy peripheral machines for support
  4. Start!

Lessons Learned

Some things to keep in mind if you decide to move forward with a similar configuration:

  • Be aware of software licensing before moving systems. Ensure the vendor provides support for testing with the license you now run.
  • Servers are often supported by peripheral services such as AV, AD, and others. Be sure for a clean environment you include those in your scripts for copying over.
  • VMRC helps a ton in Vsphere 5.5 and higher, but in lieu of this ability you will need some means to transfer files back and forth.
  • 10 NIC is the most you can attach to a VM. This will complicate things with VyOS. Keep in mind you only need the 10 networks you are using and not ALL of them at once. So A, C, E, F instead of A-F.
  • This can be replicated a thousand times over with only your capacity being the hinderance. Test Network 2-7 are okay too. Just replicate the configuration.
  • Active Directory Tombstones will bite if you are not careful. 60 days and you should plan on dumping and starting fresh.
  • Nothing stipulates you cannot migrate a Test Network to Prod. Keep in mind that in this method the copy goes both ways.
  • Systems with external interfaces will need simulators to mimic those connections. Especially important in manufacturing scenarios.
  • System load is usually a big impact point to keep in mind when testing configurations. Be sure to include ways to force system load to match enterprise. Maybe I will write a blog post on the various SQL, HTTP, SSH, etc. ways to do this with some python code..

Benefits

  • You are guaranteed to have a test network that represents your production network.
  • The on-demand nature ensures that storage and CPU cycles are only used when needed.
  • Depending on the size of the machine this can take any where from 30 minutes to 2 hours, but it is quicker than spinning a new environment.
  • Serves as a great disaster recovery test.

Updated: