I work in a small IT deparment and about a decade ago we dove into the microsegmentation world. With the news constantly covering stories about breaches, security vulnerabilities, and just the general world of hacking, moving into a server network that is microsegmented made lots of sense. It reduces attack surfaces, prevents lateral movements, and provides lots of advantages than the typical flat networks most are probably accustomed to. But it doesn’t come with its challenges and hurdles to face. I’ll cover the lessons that we’ve learned, how we’ve addressed those challenges, and achieved a working playbook to make this system work on a regular basis.
As a server engineer, I’ve spent most of my career inside of a server or workstation OS of some type and initially really didn’t understand or know how networks worked. It was just the thing that was there to make my servers talk to other devices on the network. After 10 years of living in this environment there’s a few things that quickly became essential to being sucessful in working in this type of enviroment:
- Fundamentals: Having a basic understanding of the fundamentals of technology is a must
- Documentation: Vendor Documentation becomes essential
- Logging: A good logging strategy is needed to understand what’s happening in the environment
- Practice makes Perfect: Taking a playbook from Tom Cruise’s movie in Edge of Tomorrow: Live. Die. Repeat.
Fundamentals
First and foremost, having a good understanding of the technology and general concept of how things work are very important. When you don’t know how the technology works, it becomes very confusing to understand why something is happening in the environment. For example, if ICMP is blocked from the users network to a server, then pinging the server will fail. But if port 443/tcp is opened then that will allow you to browse to a website on that server on that port even though you can’t ping it. If you don’t understand that ICMP is different than SSL traffic over port 443 this can be a confusing situation for sysadmins and engineers working in the space. A few tips to think about if you’re in an environment like this that can help you be successful.
- Get to know the environment that you’re in
- Get a network diagram or topology of the layout. Understand what stands between point A and point B. This will help you understand what hurdles you have to clear.
- Get a general understand of the type of firewalls being used in the environment.
- Are they next-gen firewalls that identify the type of traffic traversing the firewall? Or is it just based on ports being opened or closed?
- A next-gen firewall can identify the type of traffic passing over a certain port, and allow the “application” along with the port number. This extra layer of checking prevents certain malicious actors from being able to tunnel traffic over an open port that is not intended for that function. This would allow undesired actions to occur.
- Understand if ICMP is opened and pinging things are expected to work, or is it blocked and not allowed. Note: we allow ICMP internally for everything as an easy method for troubleshooting
- Understand the basics of network routing: On the initial surface, a routing problem can appear to be a firewall block. This can become a challenging issue to understand and convey to the network engineering team. If ICMP is open that helps troubleshooting as being able to ping a device means that you have a route to reach that device. If another service on another port isn’t work this makes it easier to know that a firewall is preventing that. If pinging it fails then it could mean a routing issue. Doing a trace route to review the path it’s taking is helpful to the network engineer trying to understand the issue.
- Having regular training sessions internally to make sure everyone understands the fundamentals and knows how to test for the specific problem being experienced is very important to be successful.
Documentation
Vendor documentation: Anyone that has spent time in the IT world has come to know vendor documentation. There’s varying levels of competent documentation, but every vendor should provide some type of information about their application. Included in this documentation should be network ports required for the application to function as expected. It will include the source, the destination, the port number, and the protocol it uses. Most all apps will need a default set of ports open to function as expected (DNS, NTP, Active Directory if it’s used, etc), as well as custom ports specific to the application. Knowing the traffic flow is very important to understanding what you need to request. (e.g. does the server start the communication to the laptop, or does the laptop start the communication to the server?) A small list of typical ports are (there’s dozens of others):
- 21 – FTP, File Transfer Protocol
- 22 – SSH, Secure Shell
- 23 – Telnet
- 25 – SMTP, Simple Mail Transfer Protocol
- 53 – DNS, Domain Name System
- 80 – HTTP
- 110 – POP3, Post Office Protocol
- 123 – NTP, Network Time Protocol
- 443 – HTTPS, HTTP over TLS/SSL
- 3389 – RDP, Remote Desktop Protocol
Internal Documentation: In conjunction to the vendor documentation, you should develop a good strategy for internal documentation. This allows you to help engineers working in the environment to understand what is set up and what the expected behavior is. You also have to make sure that you have a good procedure to deal with blocks and issues as they arise. We have an internal wiki that we try and document each application, the servers running that application, and the custom firewall rules required for that application to function.
Logging
At the core of all of this is having a good logging strategy in place so that everyone, not just the network engineers, are able to see the firewall or network security device logs and understand what’s happening. We use splunk as our syslog system and all users have read-only rights to the logs. This allows anyone to search firewall logs to see if attempted connections were allowed or blocked. Understanding the logs starts with making sure you have the first bullet point, a solid understanding of the fundamentals.
Practice Makes Perfect
As with anything you do, whether dieting, playing a musical instrument, or learning a new skill, practice makes perfect. When you first start down the path of microsegmentation, you have to understand that you’re probably not going to be successful right at the beginning. There’s going to be learning pains, new challenges, and a different way of thinking than you’ve become accustomed to. So going into it, you have to acknowledge that you will probably have multiple challenges you’ll have to overcome. Setting up good processes and a good framework before moving into the rollout will determine if you’re successful.
Steve's Stuff 