For the past few days we were in a war room senario because we were encountering an issue where our ERP system that sends PDF attached emails via Unix nail was experiencing a delay that caused the queue to back up significantly and preaty much piss all of our associates off . Of course everyone blamed the “network” isn’t it always lol. So back to the issue every other type of email the servers sent no delay at all only ones with the PDF attachments were seeing this delay.
Now we have some pretty sweet tools Extra Hop that we span a lot of our links though a gigamon tap. It’s collects a ton of information and always gives us a good idea if there is a poblem what it is. We also have a tool called Omni Peek , my personal favorite when we need to get down and dirty and start doing packet captures it can take 100MB capture files with millions of packets less than min to use it analysis engine to find what it deems as potienal issues.
This problem was kicking our butts all packet captures were telling us was “SMTP slow response” We rebooted and rebuilt our relay gateways and no change. We spun up Azure SendGrid to take exchange out of the picture , still had the slowness.
After days of this we found a bandaid of just send more sessions. We were doing 3 sessions a second we pumped that up to 15 and it worked but still with a delay. Then one of our network techs stumbled on to something he said I’m seeing a lot of DNS errors in Extra Hop could that be it?
Well the box over the last months has had those lookup errors and this problem was only a few days so we thought yeah it’s an issue but not the issue. At this point we were grasping at straws so we said screw it let’s remove one of our DNS server VIPs out of the picture and BAM!!! Everthing work 100% no delays. We were all stunned. Why was the nail client in Unix the only thing affected by this? Maybe it’s that sensitive??? But now it’s fixed and I was glad it was a Friday, left work early and started a few days of vacation.
Great job Will. You always had a keen troubleshooting mind, keep it up.
I had a similar situation back in the dim and misty. Packet capture revealed that the Sendmail servers were doing a reverse lookup on every destination mail gateway. At that time (and probably still to this day) many zones weren’t properly populated with PTR records, leading to long delays waiting for queries to timeout. Disabling that lookup option quickly fixed the issue. Doesn’t sound like that’s your scenario, but it’s always amusing when the Network guys have to tell the App guys how their stuff is *actually* working!