We’ve recently installed a couple of Juniper J-Series routers that
have the new JUNOS with Enhanced Services installed on them. During the
transition from our existing Linux routers, we started moving internal
subnets to the new routers, but when we moved the first subnet, we
discovered a problem with hosts that had addresses on two different
subnets. Connections would either connect for a minute and work and then
get a connection reset, or packets would come in to the server and leave
again, but then get swallowed in the ether.
I spent quite a bit of time this week reading about the security
features of the new routers, and finally came up with a solution. The
first clue was that I was getting connection reset from something on the
network, but carrying out packet sniffing on our existing routers and
the end points showed that they weren’t generating it. I eventually
found the tcp-rst option, which generates a reset packet for
any non-SYN packet that doesn’t match an existing flow session. JUNOS ES
does stateful packet inspection by creating a session when it sees an
initial SYN packet and then does filtering and routing based on that
flow session so it doesn’t have to do it for every packet. When I turned
off the tcp-rst option on the trust zone, my connection that
worked for a minute worked again only for a minute, but this time, it
just hung, rather than dying with a connection reset. This cemented the
idea that the Juniper routers were the cause.
It turned out that the problem was that there was asynchronous
routing going on. A packet was coming in to 10.0.0.1/24, but the server
was also on 10.0.1.1/24 and the default route nexthop was 10.0.1.2.
Depending on which subnet we moved depended on the resulting behavour.
If we moved 10.0.0.0/24 to the new routers, they would only see the
incoming side of the conversation. If we moved 10.0.1.0/24, they would
only see the outgoing side of the conversation. If we then think how
this would work with the session-based routing and firewalling, in the
first case, the router would see the initial SYN packet, but would never
see the returning SYN-ACK packet, and after an initial timeout, decide
the flow never established and destroy the session info, resulting in
further incoming packets to be dropped. In the second case, it would
never see the SYN packet, only the SYN-ACK. This packet wouldn’t belong
to an existing session, so would be blocked. The solution is to turn off
the SYN check, using:
[edit] user@host# set security flow tcp-session no-syn-check
After commiting that, sessions work correctly, even without the
router seeing both sides of the connection:
user@host> show security flow session destination-prefix 10.0.0.1 Session ID: 201341, Policy name: default-permit/4, Timeout: 1798 In: 192.168.0.1/61136 --> 10.0.0.1/22;tcp, If: ge-0/0/0.7 Out: 10.0.0.1/22 --> 192.168.0.1/61136;tcp, If: ge-0/0/1.7 1 sessions displayed
Sadly, I couldn’t find much information on the no-syn-check
option, so hopefully people will find this explaination useful.
on said:
Heh. Have you checked that the J-series boxes can keep up with your traffic requirements? With plain JUNOS, the stateful filter performance is pretty poor, and not comparable at all to Netfilter on similar hardware.
on said:
Just wanted to drop you a note and say thanks. We were having the exact same problem setting up to J-series boxes in a multihomed BGP environment where asymmetric flows are the norm.
I was banging my head against this problem for hours until I found this blog post and that one simple command line.
Thanks again!
on said:
Juniper’s flow solution on routers annoys me to no end. I believe that firewalls should firewall, and routers should route. We already have Juniper firewalls, and when they made the flow configuration non-optional in JUNOS it took me days to get our routers to work correctly. Now, after a year of working properly, a new BGP session broke everything again. Having dealt with this before, I figured that it was an async issue – but I’m running Juniper’s “recommended” config for disabling flow. Apparently no one at Juniper thought about async traffic.
Thanks for your post. It saved me from having to buy Cisco.
on said:
Hi
I wanted to say thanks for the post. I also have installed a multihoming bgp environment with this issue.
I spent so many time to explain my issue to the juniper support. They told me do enable mpls and do some configuration for this problem.
This simply command really solved my problems.
best regards
IGLU
on said:
Thank you very much! I used to work with Cisco, but several days ago j6350 has fallen on my head. The same issue on multihomed bgp configuration.After this magic command everything came to normal state.