After getting absolutely nowhere with Verizon Wireless technical “support” for two months, spending hours on the phone with reps who knew less about Internet Protocol than I do—which is pretty pathetic—and tearing half my hair out, I believe the problem with Verizon Wireless users having difficulty accessing the Tripawds Blogs and Forums has finally been resolved, once and for all. At least so far…
We’ve been using our Verizon Mifi to successfully manage Tripawds and the other websites we host for about a week now, which has been impossible to do for the last couple months due to packet loss I identified using traceroutes from our server. Don’t worry, I’m not going to get all technical again. Pardon my French, but this shit is above my pay grade, and the debacle deserves at least a little cussing.
Never mind what I feel like telling all those “techs” who insisted it was a “problem with our site,” or that our “IP address is bad.”
Instead, I’d prefer to put this behind us, and hop on. Thank you Jerry for teaching me to Be More Dog.
What happened and how did we fix it?
After making a stink on social media, I thought I was finally getting the attention of somepawdy at Verizon who could help. Apparently not.
Only after escalating the issue to Executive Management did I speak with someone who, “sincerely wanted to help get this fixed.” I discovered how to contact your Regional Verizon Wireless President, and didn’t expect to hear back from him directly, if at all. So I didn’t get my hopes when the woman I spoke with acted very diplomatic and promised to put me in touch with a network engineer. The next day, however, I was elated after receiving a call from a network tech at their Technical Advisory Group who actually admitted there was clearly a problem and promised to get to the bottom of it.
To make a long story short, I spent the next week on the phone working directly with a number of technicians who worked their magic while running live trace reports on my device while connected to our server. They identified a problem with packet/data optimization Verizon performs that was not working properly for certain network traffic. Lucky us!
They also apologized for the prior lack of support I had received, and we had a good laugh at some of the responses I was given during my many previous pleas for help.
This whole ordeal would be funny, if it wasn’t our sites, and Tripawds users who were being affected. If you use a Verizon Wireless device to access Tripawds—and you can read this—please let us know if you’re still having any trouble. If you know any other members who may have given up in frustration, please tell them to try again.
And if you appreciate the efforts and persistence that went into getting this resolved, please consider Supporting Tripawds! Had I given up, I’m certain the problem would remain today.
Verizon Wireless customers are having trouble accessing the Tripawds community, and VZW tech support fails to acknowledge the network routing issue causing problems. This has been going on for weeks and more than six hours on the phone with call center reps has only resulted in a total customer service failure by Verizon, continued frustration for users, and major stress headaches for me.
Tripawds members need to know that we have been doing our best to get this issue fixed, so I will explain what’s going on and how hard I’ve been trying get help from Verizon network engineers. Besides, such an epic Verizon support failure needs to be shared, if not solely for its entertainment value, then to document what I have discovered in case I ever reach anyone at Verizon who cares, or is able to help.
What Can’t I access My Server via Verizon Wireless Network?
I first noticed issues connecting to the server which hosts the Tripawds blogs and forums when using our Verizon MiFi wireless hotspot on or around March 21, 2015. We were in a remote area at the time, so I presumed that was the reason and continued to work effectively on the Tripawds site by using our HughesNet satellite internet connection. After relocating four times across three states over four days, I was still unable to access our site using the Verizon MiFi, when I had no trouble doing so via HughesNet.
I could use the MiFi to easily browse other websites. Only when attempting to connect to our site, did the Verizon connection timeout. Meanwhile, certain Tripawds members who also happen to be Verizon customers started reporting similar issues.
Thinking there might be a problem with the server, I opened a support ticket with our data center at Peer 1 where we manage our own dedicated server. They were the first to identify serious data loss on the VZW network, prohibiting certain Verizon customers from being able to use our website…
“I did a mtr from your server’s gateway to the destination ip 70.196.205.9 and I am seeing packet loss on verizon’s end (hop 9). If you want us to send an email to Verizon; we could. However please note that since we are not directly peered or their customer, there’s a high possibility that they will not respond back to our email. Please can you ask your client to open a ticket with Verizon regarding the packet lost.” —Peer1 Network Analyst
My Traceroute Results from Server to Verizon MiFi
I’ll be the first to admit that I have a fairly novice understanding of network routing. But let me explain what this means to anyone less technical than I, like the Verizon tech support reps I have been dealing with for instance.
What does mtr mean and what does it do?
The “mtr” stands for My Traceroute. This is a network diagnostics tool run via command line interface, in this case from our server which has a specific IP address on the internet. The results above show nearly 50% packet loss at a hop on the myvzw.com network backbone. Note how the pings (shown at left) never reach me, or my IP address at the time when using my Verizon MiFi.
Compare that to these results from an mtr I personally ran from our server to my IP address when connected via our HughesNet internet connection…
MTR results from server to my HughesNet ip address
No packet loss, and that last line is my IP address, clearly proving that the connectivity problem is specific to the Verizon network, not a problem with our server. Thus began my journey through tech support hell with Verizon call reps, where the frustration continues.
Before I get into the hilarity that ensued when trying to get help from Verizon support, I’ll share further diagnostic measures we took to confirm this was not an issue with our website, the server, or my MiFi device.
First, I wanted to confirm this same issue was the reason other Tripawds members were also having trouble. So I asked those reporting problems, who also use Verizon Wireless, to provide their IP address…
mtr results from server to Massachusetts Verizon IP address
Surprise! This member in Massachusetts uses Verizon and is unable to connect to Tripawds.
Data loss on Level 3 network for VA Verizon user.
And, this member in Virgina is continually having trouble navigating the forums or posting in the chat room.
So, you may be asking, “Wait Jim, what’s that ‘Level3’ all about? I thought you were blaming this on Verizon.” Well yeah, I am. A supervisor actually mentioned being unable to ping our server from “Level 3” after one of my lengthy support calls, and repeated representative escalations. Well, duh! That’s exactly what I’ve been reporting for the past few weeks now.
Besides, if Level3 is not a Verizon network, they are directly peered to it, and I am a customer of Verizon, not Level 3! But I digress.
Uhh… what’s a ping?
I continued to troubleshoot this issue by “pinging” our server from both networks I have available for accessing the web. The “ping” is a simple shell command often used for network diagnostics to determine if a certain point on the internet can be reached from your connection. In our case, I pinged the Tripawds server from both our Verizon MiFi and HughesNet modem. Here’s what I found…
Comparison of server ping results via different networks.
Notice how I have no problem communicating with our server via our HughesNet connection. That is our server quickly responding to confirm each ping was received. When attempting to do the same via our Verizon MiFi, each ping request merely times out. Sigh…
At this point, over a week ago, I had consulted with our data center, my server manager, and a number of internet experts and very technically adept friends. We all agreed that the problem lies in the network, there is no problem with my server or website configuration. And my MiFi is working fine.
So, about that “entertainment factor” I referred to earlier…
Does anyone at Verizon Support Understand Networking?
This support debacle is not just funny, it is pathetic. It might be amusing, if it wasn’t my server, and my customers who are are being impacted by the problem. I just can’t believe that I know more than every Verizon tech support supervisor I have spoken with. The following are just a few highlights from the hours I have spent escalating calls only to have two support tickets closed with unacceptable resolution.
This is a text message I received from Verizon after a supervisor told me the issue had been “resolved” followed by a screenshot of what I had spent more than an hour explaining to various reps. Why a text? Yeah, that’s funny in itself. A rep at the Verizon call center in Idaho told me they do not have email. Yes, an internet support center that cannot email. But wait, it gets funnier. The rep also told me they do not even use Verizon services at their location!
Frustrated, I had to call back and find out how they could possibly come up with this “resolution” for the issue I’ve reported and explained in detail multiple times. Get this: after another couple hours speaking with various reps, and their supervisors, Verizon told me “your IP address is bad.”
Funny Thing 1: Verizon Support closes both of my tickets, after determining my IP address is invalid by entering the address into a web browser and getting a page that says website unavailable. 😕
Anyone with a basic understanding of Internet Protocol or server management knows that this page actually proves the IP address is valid and the server is doing its job. We have multiple websites and 1,000+ subdomains configured at this IP address. How would a server know which one the user wants to visit? Here is further proof that our server is configured properly, with multiple domains at the IP address.
Enough said. Back to Verizon’s supposed “resolution” to our problem…
Funny Thing 2: Not being able to ping the IP address via the Level 3 network? That is exactly my point!
Funny Thing 3: I was never notified that either of my support request tickets had been “resolved” even after speaking with one supervisor who actually sounded like she may be able to help by discussing the problems with engineering. She promised that I would receive a text message allowing me to communicate with her directly, bypassing the multiple timely escalation requests every time I called.
She downright lied, or doesn’t even know how her own support system works, because when I replied the message, it came right back to me.
Funny Thing 4: I call back to express my concerns once more, and explain the problems all over again. I press the option for “tech support” so I can speak with a human. Seriously, after 20+ minutes on the phone with a rep who clearly had no clue what I was talking about, I am told, “OK, let me connect you with technical support.”
“I thought this was tech support!” I say
“No, this is customer service,” she says. Service? Not!
Various Funny Things: I did my best to remain calm, but after a few calls, I knew the drill. I would begin each call by screening the support rep to see if they know what I was talking about.
After trying to explain to one rep that this was not a problem with my device, but a network backbone routing issue, I was transferred and told…
“Please hold, we’ll be happy to help you get your backphone working.”
Another time, after clearly describing the packet loss at a hop on the Verizon network, the rep said
“I’m sorry to hear about your package loss, let me transfer you to someone who can help with that.”
Still, no help. So I turned to social media…
Apparently Verizon handles calm requests for help on Facebook by deleting posts to their page, without reply.
On Twitter @VerizonSupport was actually quite responsive with a reply suggesting I follow and send a direct message. We replied back and forth via private messages immediately. Great! or so I thought…Sigh…hoping @VZWSupport might be as responsive, I send various messages and screenshots. It’s not easy describing such technical issues in 140 characters or less, but I was polite and clear providing as much detail as possible. After waiting almost 48 hours, I get this reply to my message with a screenshot clearly showing network data loss…
Seriously? Security setting? On who’s device? This is not just my problem! It is with Verizon users all over the country. I’ve asked them to clarify “security setting” since there are no IP blocks on our server for Verizon users. Could Verizon be blocking the IP on their end? I have yet to hear a reply.
Another Funny Thing: The link they sent me is for instructions about how to use my MiFi and reset its admin password.
So here we are, weeks later with nobody willing or able to help.
Where is my Support Hero?
Please, somebody. Tell me I’m wrong. I would love for someone to tell me what’s going on and how I can fix it. That person clearly doesn’t work in technical support at Verizon. I have been told, “reporting this issue again will not help” and that I need to contact my web host. They even kindly told me who that was, as if I didn’t know!
If you know anybody at Verizon who may be able to help, please forward the link to this post!
If you can help, please leave a comment with any suggestions.
If you agree with anything I’ve said, or have had similar experiences dealing with Verizon please share this link.
And if you are a Tripawds member, please know that we’re trying desperately to et this issue resolved. Sadly, if you’re experiencing this issue, you likely won’t be able to read this. 🙁
UPDATE: April 20, 2015
Big surprise. Verizon Wireless has now closed the third support ticket about this issue, insisting that is is still “a problem with the site.”
Furthermore, we have received confirmation from network analysts that the packet loss we have identified at VZW network hops is being caused by ICMP throttling at the network nodes.
ICMP Rate Limiting is a common practice to alleviate network congestion by prioritizing “more important” traffic. How do I explain to Tripawds users that their beloved pets are not important enough for anyone at VZW support to help?
UPDATE: May 10, 2015
Finally. After getting nowhere with Verizon Wireless technical “support” for nearly two months I believe this problem has actually been resolved! At least so far…After I escalated the issue to Executive Management, I was able to work directly with network techs who actually understood me, and identified a problem with packet optimization that wasn’t working properly for everybody. Persistence pays off!
