Little Server Guy -
Info from a Guy Trying to Run a Little WS2003 Server on the Cheap
Multi-Homing
From the first second I considered hosting my own web sites, I envisioned running
multiple web sites on the same server. After all as one of MS's KB articles says
(sorry I've lost the link), this has all kinds of advantages. And it seems simple
enough...
Too bad they don't tell you how to do it. I'm going to tell you what I know, and I'm sure I'll
hear lots of upgrades. That's ok, at least it's a dialog.
As a start, I found some technet documentation on TCP/IP in server 2003, and I want
to include this link, even though I haven't studied it yet. http://technet2.microsoft.com/WindowsServer/en/library/8032dd80-9d51-4ad7-8b57-5267d61f1b411033.mspx?mfr=true
Windows Server 2003 TCP/IP Implementation Details
Looking at this, I want to say up front that I am a little guy, running a little
server. I am NOT running a domain controller on my server, and I am not running
DHCP on my server. I do have a small lan with a couple DHCP devices, and my server
is on my lan, but these DHCP devices get their address from a router and not my
server. My server has 3 NICs: one connects to my LAN router, and two connect to
my connection to the internet where I have 5 public IPs. (Sadly, these plugs sit
right next to each other in the SMC "thing" that Comcast provided me, and
I can't explain to you how traffic moves inside this SMC thing. But I digress. That
swamp is reserved for another page later.)
Two Approaches: Multiple NICs or Multiple IPs on one NIC
There are two ways to make this happen. There is a knowledge base article that dates
from Server 2000 days that addresses this, if you care to read it - I suspect it
is still valid thought I can't guarantee it: http://support.microsoft.com/kb/157025/en-us
MultiHomed Computers
Multiple IPs on one NIC (Network Interface Controller?/Card?)
You can configure a "Network Connection" to have more than one IP address. To do
this, you open the properties of the connection, select "Use the following
IP address", then click the "advanced" button. Under "IP addresses" you can click
"Add" to assign additional IPs to the connection. "Network Connections", to the
best of my understanding, are created by windows for any physical network interface
the system detects, and if this if correct, then you are asking windows to operate
that network connection using multiple IPs. There is also, on the advanced
dialog, a way to configure multple default gateways for that single connection.
I'll address default gateways a little later.
What does that mean? The last paragraph sounds like the reference books I have on
Server 2003. They tell you what you can see for yourself but not what happens when
you configure what you see. Truth is, I don't know. My guess is that windows will
register the MAC address of the NIC with all of the multiple IPs you configure,
so that the router on the subnet will send traffic for any of the IPs to that NIC.
I have not tried this, and I don't know if this is acutally what it does, how it
works, or how well it works.
Multiple NICs
Being a chemical engineer by training, it never occured to me to mix the traffic
for two web sites on one wire - sort of like using one pipe for hot and cold water?...
OK, I admit some bias to seeing things physically. After all, won't those poor electrons
get all mixed up about where they're going?
This is the configuration I've struggled with. With this configuration, you'll
have more than one physical NIC, each with an ethernet connection, and thus more
than one network connection in windows. Each connection can be configured to have
it's own IP, default gateway, and DNS servers.
"Same-Subnet" Complications
While this seems clean, there is a problem. In my little business, I have 5 public
IP address, and they belong to a /29 network, which means that they are all on the
same subnet. I happen to have Comcast business internet service, and they give me
an "SMC" combined cable modem/router with 4 ethernet ports on it. This monstrosity
does what it does, but it is unlike standard network device I know of. Anyway, if
I configure two network connections in my server to have 2 of my public IPs, and
plug the corresponding NICs into this SMC thing (for lack of more precise term),
then both NICs see the www directly with no firewalling or other filtering in the
SMC. However, the SMC will pass traffic within my 5 IP subnet directly, as if it
were a switch. Neglecting security for the moment, I could try to run my server
this way, and it works.... at first. BUT, there is a limitation in having two NICs
within this server that since traffic moves between these two NICs (effectively
a LAN), only one of the interfaces may transmit at a time. Apparently they can listen
at the same time, but like any other two network entities, they have to wait a turn to broadcast.
This may or may not be a problem, I'm not sure. There is another knowledge base
article that speaks to this, reviewed in Jan 2006 but again applying to Server 2000:
http://support.microsoft.com/kb/175767/en-us
Multiple Adapters
.
Default Gateways
Every network connection in windows has a default gateway. What is it?
Well, this is muddy. YOu can configure more than one per connection, and also have
more than one connection. So what does it mean to be a "default"? Well, it isn't
really a default. It is used by windows to create a route in the routing table for
the computer. This is described in the knowledge base article cited above, which
again does not state that it applies to server 2003, but only up through server
2000: http://support.microsoft.com/kb/157025/en-us
MultiHomed Computers
. There is says that "Every NIC that has a default gateway adds a 0.0.0.0 route
(default route) to the routing table." I can verify this - it does work this way.
However, having this route in the table does not seem to be enough unless the NIC
also has the default gateway configuration that matches. (Let me digress or jump
ahead here for completeness: Configure a default gateway on a NIC; windows adds
the 0.0.0.0 route; the NIC works; but if windows deletes the default gateway spec
from the NIC configuration, the NIC stops working even though the route is still
in the routing table. This is my experience with one of my NICs ceasing to work,
which I believe (not positive) is due to "dead gateway detection" which I'll address
below)
It does not say what happens if you configure multiple default gateways on
one NIC. However, I believe that windows will add a 0.0.0.0 route for every default
gateway that is configured.
What is a 0.0.0.0 route? In simple terms, it is like a catch-all; if there is no
known way to get to a particular IP, it is considered to be "out there" on the www
and the 0.0.0.0 route tells the server where to hand it off to get it out on www.
Interface Bindings
And you thought this was like plumbing a faucet? Ha.
Each route in the routing table has a little more to it: an interface. "Interface"
here means NIC. You have multiple NICs; i.e. multiple pipes to send packets out
from the server. Each entry in the routing table not only defines the destination
to send a packet to, but it also says which pipe to send it out on: the "interface"
as it's called in the routing table and the route untility commands (or NIC as we tend to call it otherwise).
So when windows creates the 0.0.0.0 route for the default gateway on the first NIC,
that route specifies that the packet goes out on the "interface" which is that NIC.
I won't even begin to speculate how this is done with multiple IP's per NIC...
So in my server, with two NICs that point to two of my public (www) IPs, if I configure
a default gateway on both network connections, windows creates in my routing table
two 0.0.0.0 routes - "default" routes - and one is bound to each of my NICs
and points to the default gateway for that NIC routed out of the server on that
NIC. Sounds nice, eh? That's the kind of setup that would make a chemical engineer
happy...
Metrics
There is one more little complication on these routes: the metric. Metric is an
integer and it helps the server break ties when it is choosing a route. Each route
has a metric value. If the server needs to
go to 0.0.0.0, it will choose the 0.0.0.0
route with the lowest metric. At least that is what the basic stuff I've read on
TCP says.
Windows wants to set this metric for you, by trying to figure out what route works
best. I've read some about this, but I don't have the links in mind now, so I'm
recalling. It tries sending something and the fastest route gets the lowest metric.
Or it counts hops and the lowest hop count gets the lowest metric. You can override
this in the "advanced" dialog after you've selected "Use this IP address" in configureing
the network connection. IN that case the route metric will not change.
Maybe another time, in another life I'll have time to figure out what metrics do
in this scenario. About all I can say now is that leaving them on "automatic" gives
windows the freedom to change them, which might change how it tries to route outbound
traffic. ANd since my big problem has been getting a stable system, I don't see
how this could do anything but hurt things. So below I recommend you set these metrics
to be fixed with a value of 1.
The Hard Reality of Multiple NICs -Dropping Off Line
However, in my multiple NIC server, things are not this clean. In my server I have
two NICS for my public IPs. Just for completeness, I've configured them both to
point directly to the internet, so they are on the same subnet; and I've subsequently
changed the network using an intervening router to put them on different subnets,
using NAT in the router. In other words, in one case on the same subnet; in the
other case different subnets. The behavior I see is the same for both cases, so
I think it is not related to them sharing the subnet.
In all cases, I've configured each NIC to have one IP, and configued that NIC with
a default gateway that works for that IP. When the NICs are on the same subnet,
those default gateway IPs are the same IP address. When they are on different subnets,
they each have a default gateway IP that is on the subnet of the NICs IP (this isn't
an option. The default gateway points to the router which is the only way to get
out of the subnet, so the default gateway HAS to be on the same subnet as the IP
of the NIC).
When this boots up, a couple things happen. Each NIC takes its configuration and
this can be view using ipconfig at the command line.
Since both have a default gateway
defined, both show a default gateway as part of the network connection info, right
along with the NICs IP and Subnet mask.
The second thing is that both network connections cause a 0.0.0.0 route to be created
in the routing table. You can see this by using "netstat -r" at the command line,
or the equivalent "route print" at the command line. MS advertises these are identical
and the output appears identical.
With this, all seems well, and both NICs come up and serve their IPs. I have two
IIS web sites, each pointed to one of these incoming IPs. Page requests come in,
and pages are served back out.
Fine, until.... who knows when, but when it happens (time elapsed from 20 min to
5 days in my experience) one of the NICs completely ceases to function. Examining
everything I can find to look at, including "ipconfig/all", "Netstat -r", and "arp
-a" (a table linking IPs to the MAC address of the physical ethernet hardware),
I can see only one change: for the NIC which has stopped, the default gateway is
now gone - it simply shows blank.
Dead Gateway Detetion
I have been wrestling with this nonsense for nearly a month now (now being Oct 2006,
fwiw), and my first unsuccessful attempt to fix this was based on the metric specification
on the advanced IP dialog. I'm going to cite this reference even though I don't
find that it works, just for completeness. This is posting somewhere that was made
when someone converted from 2000 to 2003, I believe. The link is
http://www.commodore.ca/windows/windows_2003_upgrade.htm
Windows 2000 Domain to Windows 2003 Upgrade Procedure
. Skip to the bottom of this page to the section: Multihomed Server Routing Problems.
I actually didn't understand this completely and I did a part of what is suggested
here, but I still don't believe this has the complete answer. It does match the
knowledge base recommendation in the second berger dot in the default gateway knowledge
base article
MultiHomed Computers
I referenced above. The 2000 to 2003 page says " We used to try and run multihomed
servers with 2 "default gateways" - every now and then the server would drop off
the internet. A server can only have one default gateway. Its basically a dumping
ground for any traffic it doesn't know where to route. The correct way (and now
enforced by win2003) is to add static routes."
However, I see on my server a NIC stops working even though the routing table has
still has the route defined. Thus, this page suggests a different way to get the
route into the table (making a "static" or permanent entry into the table by hand
rather than allowing windows to create it when it boots). However, I see that my
NIC stops working when it's default gateway specification is lost. I didn't exactly
follow the recommendation on the 2000-2003 server page, but I don't think it will
do for my case because it won't stop the default gateway spec from diappearing on
one of the NICs. This page also discusses the automatic metric. I don't think the
recommendation here will work, in fact I think its not relevant to their case of
two NICs on separate subnets. The metric would never come into play there because
the routes generated by windows for the two NICs will never point to the same places
- except the 0.0.0.0 routes resulting from the default gateways, and I suspect that
application interface binding (maybe I'll get to that later) will make that irrelevant.
I think the issue here is
dead gateway detection. "Dead" must be
understood here to have its own meaning - here "dead" simply means "I'm sick of
what you're doing so I'm banishing you from service forever," where forever means
until a reboot.
Dead gateway detection is part of RFC 816, which means it isn't really MS's fault.
Ahem... everything is MS's fault, isn't it? For yet another out-of-date knowledge
base article on this, see http://support.microsoft.com/kb/171564
TCP/IP Dead Gateway Detection Algorithm
which dates all the way back to NT. Well, heck, nothing's changed since
then, right?
Let's hope not for this subject.
I don't know what "failing" means in this context, but apparently, when 25% of the
active TCP connections fail, the default gateway being used is discarded in favor
of the "next" default gateway. This is intended, I think, to provide robustness
in the face of network failure, but it makes some assumptions that might not be
good ones. What if my little web site has only one visitor, from say far way, and
the packets sending back the requested web page don't make it. One failure of one
is more than 25%, so dump the gateway. Huh?
This gateway failover
appears to be happening on my server.
I don't know how to document it or identify it. Perhaps it is an event in the logs?
I don't know, and haven't seen anything from MS to explain how to ID this happening.
Anyway, I sketched earlier, what I observe is that one NIC stops working completely.
The only configuration change I can see is that the default gateway specification
is now blank on the NIC that stopped.
OK. So dead gateway detection is intended to deal with network failure. This isn't
a concern for me. It is triggered by conditions which, for my little sites, can
happen pretty easily, and
nothing to do with the functioning
of my network connections. And it complete shuts down my web site. Can you spell
"No thanks, RFC816?"
Application Binding to Interfaces
Metrics are defined as a priority system for selecting among possible routes. The
lower metric is chosen in that case. however, my observation is that IIS will not
reroute its traffic. Dead gateway detection takes away the default gateway for one
of my public NICs. Reading the MS stuff (sorry, no specific references to offer,
this is a muddy recollection of things I've read in the KB and technet) says that
the default gateway is switched when a gagateway is detected as dead. To me this
seems to suggest that the default route (the 0.0.0.0 route in the routing table
which routes over that interface (again, read NIC)) should be deleted, or it's metric
increased so it isn't used. I don't see either. What I see is a change only in the
NIC default gatway spec, and that isn't
changed, it's
deleted. This seems to give lie to MS's contention
that a "default gateway" belongs to the system and not the NIC.
Moreover, when that default gateway spec is deleted, there are still valid routes
to the internet in the routing table that can be used by ISS to service that web
site traffic. This does not happen. I don't know if the IIS is still receiving the
inbound requests and just can figure out how to send the pages in response, or if
the inbound traffic stops. I suspect, from my monitoring, that inbound traffic is
stopped. This may mean that windows is somehow broadcasting (speculation, based
on my limited understanding of this) that the route to that NIC is no longer available.
The answer to this may lie in a concept alluded to in some technet or KB stuff I
read. They talk about 'application binding" to interfaces. In other words, an application
speaking tcp/ip may limit its traffice to a specific network connection/interface/NIC.
Sorry, this is ugly but multiple terms are used in different contexts. If an application,
like IIS, binds it's traffic to a particular interface, and if windows pronounces
that interfaces default gateway to be "dead" and disables it, then that application
has no connection to the outside any more. This may be part of the issue I am seeing,
where my server has 3 functioning (hardware-wise), valid routes to the internet
available, yet completely stops all traffic on one of it's NICs.
The Fix: Disable Dead Gateway Detection
It seems so simple now... Why did it take a month to figure it out? Because nobody
wrote anything down about it, even if they understood it. Thus, this page.
The solution is disable dead gateway detection. My experience says that you must
still fly in the face of the knowledge base recommendation and configure default
gateways on each network connection. Disabling this is described in the context
of hardening the TCP/IP stack against denial of service attacks.
Finally
I find a knowledge base article on this that is actually current for server 2003:
http://support.microsoft.com/kb/324270/en-us
How to harden the TCP/IP stack against denial of service attacks in Windows Server 2003
. The second berger dot under TCP/IP registry values shows how to do this. Just
in case that article moves, I'll put the regedit navigation path here: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\EnableDeadGWDetect
. This should be created as D_word and set to 0 to disable this automatic function.
My Multihoming Recipe for Windows Server 2003
Here's what I know:
1. Get a computer.
2. count the ethernet plugs on it. (Mine has one.)
3. Count the number of public web sites you want to host on separate public IP addresses
(for me, 2). Add 1 for your lan to get the number of plugs you need.
4. Whatever the difference is, buy hardware and install it so you have the number
of plugs you need. I bought 2 PCI ethernet "adapters" and installed them.
5. Configure the IP on each "network interface" so that each one is on it's own
subnet. One should be your lan. Why it's own subnet? If I knew, I'd be a network
consultant. I DO know, however, from another unnamed MS KB document, that according
the rules of ethernet, only one interface can talk on a subnet at a time. Actually
I think it can work ok on the same subnet, but since I'm sugesting you use a router
for a firewall, you might waas well map them to their own subnets.
6. Configure the IP on each network interface to have a fixed "metric" of 1.
7. Ignore the message from the configuration that tells you not to configure multiple
default gateways. Configure each with a default gateway of its default gateway,
whether the same or different from one another.
8. Now you have to figure out to present the traffic from your first public IP to
one interface, and the traffic from the other to a second. Let's look deeper for
a sec. I'm going to suggest you do just what I did, which is isolate each server
NIC behind a separate firewall router.
Intially I resisted spending money for a firewall. But since I realized I should
reroute my public IP traffic onto unique subnets, I was going to need one or more
routers anyway, so I caved on this. After spending probably too much time fishing
around and reading, I decided that Netgear's firewall routers seemed to be about
the right sophistication for me. They are more tuned to an inbound public
facing network (like a web server) than a home router, but not much more expensive. I ended
up ordering two refurbished Netgear FVS318 firewall routners from justdeals.com
and paid about $65 for both delivered. They each have an 8 port switch where I only
need one "lan side" port. They have SPI filtering and can send email reports on
DOS attacks and email their logs as well. They're metal boxes that stack, unlike
most of the retail home stuff.
Configure
the WAN side of one router to be static using one of your public IPs,
adding the gateway and dns servers. Assign the LAN side to a completely unique
subnet. Thereare lots of private IPs, so
I chose 10.1.0.0. I used the whole 256 subnet since I don't need to economize: a subnet mask of
255.255.255.0 (this means all the final octets from 0 through 255 belong to this
subnet. No, I'm not going to use them all). The router's identity on the LAN is
the customary one: 10.1.0.1 I don't plan to hang any other devices on this subnet - it is devoted to the web
site. So I turned off the DHCP server on the router's LAN side. I then configure
the servers NIC to have the IP 10.1.0.2, with the subnet mask the same as the router's lan side: 255.255.255.0,
and the router itself is the default gateway for this NIC:
10.1.0.1 . Yes, my experience, above, says that each network connection (NIC) must
have it's own default gateway. Then use the same DNS as the router itself uses.
Cable the NIC to one of the LAN ports on the router.
The firewall is completely closed to incoming traffic by default. If you have only the web site (and no secure site) on this IP, map port 80 incoming on the prublic IP to port 80 on the server IP 10.1.0.2. Now http requests to the public IP are
passed through the SPI and DOS checks on the router and land on the server NIC on
port 80. If you are running a secure server or mail server or ftp server, you have
to also open those ports to pass through. (This is technically NAT, since you are
simply translate the nework address and not the ports on this traffic. If you were
to map port 80 to port 8080, that would be PAT: port address translation)
Now repeat this process using the second router and the second public IP,
giving this router it's own unique subnet, like 10.2.0.0, and repeat all
the other router and interface configuration using that subnet.
9. Now you need web sites. SO make them. On server 2003 you'll end up with a default
site, an admin site, and then I created two more for my two public web sites. You have
to configure the IP address they'll listen to in IIS Admin. The two public
sites will listen to 10.1.0.2 and 10.2.0.2. If you have a LAN and another NIC for
your lan, you can assign the default site and the admin site to listen to those
IP's only.
10. Finally disable "dead gateway detection", as described above. If you don't,
expect one of your NICs to simply stop working for no obvious reason.
If I'm right, this will give you a stable server and a reasonably secure interface
to the internet for your web sites and other public services.
Related search terms:
multi-homed server; multihomed server; multihoming; multiple public IPs; multiple
public web sites; multiple NICs; multiple network connections; multiple network
interfaces; more than one web site; multiple web sites;
default gateway; default IP gateway; default internet gateway; dead gateway detection;
defining multiple default gateways; multiple default gateway error;
network failure; NIC failure; NIC dropping offline; IP address failing; web site
failing; tcp communication failing; NIC stops working; network communication stops;
network interface stops working; network stops communicating
same subnet; disjoint networks; disjoint subnets;
Comments? Combine the word "richard" with the words "server guy" with no spaces
or punctuation and stick it in front of the domain this page is hosted on. That
should get an electonic message to me.
(c) 2006 Richard Skerritt