network – Behind the Scenes

Verizon Wireless Technical Support Fails to Help Tripawds Users

Verizon Wireless customers are having trouble accessing the Tripawds community, and VZW tech support fails to acknowledge the network routing issue causing problems. This has been going on for weeks and more than six hours on the phone with call center reps has only resulted in a total customer service failure by Verizon, continued frustration for users, and major stress headaches for me.

Tripawds members need to know that we have been doing our best to get this issue fixed, so I will explain what’s going on and how hard I’ve been trying get help from Verizon network engineers. Besides, such an epic Verizon support failure needs to be shared, if not solely for its entertainment value, then to document what I have discovered in case I ever reach anyone at Verizon who cares, or is able to help.

What Can’t I access My Server via Verizon Wireless Network?

I first noticed issues connecting to the server which hosts the Tripawds blogs and forums when using our Verizon MiFi wireless hotspot on or around March 21, 2015. We were in a remote area at the time, so I presumed that was the reason and continued to work effectively on the Tripawds site by using our HughesNet satellite internet connection. After relocating four times across three states over four days, I was still unable to access our site using the Verizon MiFi, when I had no trouble doing so via HughesNet.

I could use the MiFi to easily browse other websites. Only when attempting to connect to our site, did the Verizon connection timeout. Meanwhile, certain Tripawds members who also happen to be Verizon customers started reporting similar issues.

Thinking there might be a problem with the server, I opened a support ticket with our data center at Peer 1 where we manage our own dedicated server. They were the first to identify serious data loss on the VZW network, prohibiting certain Verizon customers from being able to use our website…

“I did a mtr from your server’s gateway to the destination ip 70.196.205.9 and I am seeing packet loss on verizon’s end (hop 9). If you want us to send an email to Verizon; we could. However please note that since we are not directly peered or their customer, there’s a high possibility that they will not respond back to our email. Please can you ask your client to open a ticket with Verizon regarding the packet lost.” —Peer1 Network Analyst

My Traceroute Results from Server to Verizon MiFi

I’ll be the first to admit that I have a fairly novice understanding of network routing. But let me explain what this means to anyone less technical than I, like the Verizon tech support reps I have been dealing with for instance.

What does mtr mean and what does it do?

The “mtr” stands for My Traceroute. This is a network diagnostics tool run via command line interface, in this case from our server which has a specific IP address on the internet. The results above show nearly 50% packet loss at a hop on the myvzw.com network backbone. Note how the pings (shown at left) never reach me, or my IP address at the time when using my Verizon MiFi.

Compare that to these results from an mtr I personally ran from our server to my IP address when connected via our HughesNet internet connection…

mtr results from server to hughesnet ip address — MTR results from server to my HughesNet ip address

No packet loss, and that last line is my IP address, clearly proving that the connectivity problem is specific to the Verizon network, not a problem with our server. Thus began my journey through tech support hell with Verizon call reps, where the frustration continues.

Before I get into the hilarity that ensued when trying to get help from Verizon support, I’ll share further diagnostic measures we took to confirm this was not an issue with our website, the server, or my MiFi device.

First, I wanted to confirm this same issue was the reason other Tripawds members were also having trouble. So I asked those reporting problems, who also use Verizon Wireless, to provide their IP address…

mtr results from server to Massachusetts Verizon IP address

Surprise! This member in Massachusetts uses Verizon and is unable to connect to Tripawds.

mtr results showing Level 3 network data loss — Data loss on Level 3 network for VA Verizon user.

And, this member in Virgina is continually having trouble navigating the forums or posting in the chat room.

So, you may be asking, “Wait Jim, what’s that ‘Level3’ all about? I thought you were blaming this on Verizon.” Well yeah, I am. A supervisor actually mentioned being unable to ping our server from “Level 3” after one of my lengthy support calls, and repeated representative escalations. Well, duh! That’s exactly what I’ve been reporting for the past few weeks now.

Besides, if Level3 is not a Verizon network, they are directly peered to it, and I am a customer of Verizon, not Level 3! But I digress.

Uhh… what’s a ping?

I continued to troubleshoot this issue by “pinging” our server from both networks I have available for accessing the web. The “ping” is a simple shell command often used for network diagnostics to determine if a certain point on the internet can be reached from your connection. In our case, I pinged the Tripawds server from both our Verizon MiFi and HughesNet modem. Here’s what I found…

server ping results comparison — Comparison of server ping results via different networks.

Notice how I have no problem communicating with our server via our HughesNet connection. That is our server quickly responding to confirm each ping was received. When attempting to do the same via our Verizon MiFi, each ping request merely times out. Sigh…

At this point, over a week ago, I had consulted with our data center, my server manager, and a number of internet experts and very technically adept friends. We all agreed that the problem lies in the network, there is no problem with my server or website configuration. And my MiFi is working fine.

So, about that “entertainment factor” I referred to earlier…

Does anyone at Verizon Support Understand Networking?

This support debacle is not just funny, it is pathetic. It might be amusing, if it wasn’t my server, and my customers who are are being impacted by the problem. I just can’t believe that I know more than every Verizon tech support supervisor I have spoken with. The following are just a few highlights from the hours I have spent escalating calls only to have two support tickets closed with unacceptable resolution.

This is a text message I received from Verizon after a supervisor told me the issue had been “resolved” followed by a screenshot of what I had spent more than an hour explaining to various reps. Why a text? Yeah, that’s funny in itself. A rep at the Verizon call center in Idaho told me they do not have email. Yes, an internet support center that cannot email. But wait, it gets funnier. The rep also told me they do not even use Verizon services at their location!

Frustrated, I had to call back and find out how they could possibly come up with this “resolution” for the issue I’ve reported and explained in detail multiple times. Get this: after another couple hours speaking with various reps, and their supervisors, Verizon told me “your IP address is bad.”

Funny Thing 1: Verizon Support closes both of my tickets, after determining my IP address is invalid by entering the address into a web browser and getting a page that says website unavailable. 😕

Anyone with a basic understanding of Internet Protocol or server management knows that this page actually proves the IP address is valid and the server is doing its job. We have multiple websites and 1,000+ subdomains configured at this IP address. How would a server know which one the user wants to visit? Here is further proof that our server is configured properly, with multiple domains at the IP address.

Enough said. Back to Verizon’s supposed “resolution” to our problem…

Funny Thing 2: Not being able to ping the IP address via the Level 3 network? That is exactly my point!

Funny Thing 3: I was never notified that either of my support request tickets had been “resolved” even after speaking with one supervisor who actually sounded like she may be able to help by discussing the problems with engineering. She promised that I would receive a text message allowing me to communicate with her directly, bypassing the multiple timely escalation requests every time I called.

She downright lied, or doesn’t even know how her own support system works, because when I replied the message, it came right back to me.

Funny Thing 4: I call back to express my concerns once more, and explain the problems all over again. I press the option for “tech support” so I can speak with a human. Seriously, after 20+ minutes on the phone with a rep who clearly had no clue what I was talking about, I am told, “OK, let me connect you with technical support.”

“I thought this was tech support!” I say

“No, this is customer service,” she says. Service? Not!

Various Funny Things: I did my best to remain calm, but after a few calls, I knew the drill. I would begin each call by screening the support rep to see if they know what I was talking about.

After trying to explain to one rep that this was not a problem with my device, but a network backbone routing issue, I was transferred and told…

“Please hold, we’ll be happy to help you get your backphone working.”

Another time, after clearly describing the packet loss at a hop on the Verizon network, the rep said

“I’m sorry to hear about your package loss, let me transfer you to someone who can help with that.”

Still, no help. So I turned to social media…

Apparently Verizon handles calm requests for help on Facebook by deleting posts to their page, without reply.

On Twitter @VerizonSupport was actually quite responsive with a reply suggesting I follow and send a direct message. We replied back and forth via private messages immediately. Great! or so I thought…Sigh…hoping @VZWSupport might be as responsive, I send various messages and screenshots. It’s not easy describing such technical issues in 140 characters or less, but I was polite and clear providing as much detail as possible. After waiting almost 48 hours, I get this reply to my message with a screenshot clearly showing network data loss…

Seriously? Security setting? On who’s device? This is not just my problem! It is with Verizon users all over the country. I’ve asked them to clarify “security setting” since there are no IP blocks on our server for Verizon users. Could Verizon be blocking the IP on their end? I have yet to hear a reply.

Another Funny Thing: The link they sent me is for instructions about how to use my MiFi and reset its admin password.

So here we are, weeks later with nobody willing or able to help.

Where is my Support Hero?

Please, somebody. Tell me I’m wrong. I would love for someone to tell me what’s going on and how I can fix it. That person clearly doesn’t work in technical support at Verizon. I have been told, “reporting this issue again will not help” and that I need to contact my web host. They even kindly told me who that was, as if I didn’t know!

If you know anybody at Verizon who may be able to help, please forward the link to this post!

If you can help, please leave a comment with any suggestions.

If you agree with anything I’ve said, or have had similar experiences dealing with Verizon please share this link.

And if you are a Tripawds member, please know that we’re trying desperately to et this issue resolved. Sadly, if you’re experiencing this issue, you likely won’t be able to read this. 🙁

UPDATE: April 20, 2015

Big surprise. Verizon Wireless has now closed the third support ticket about this issue, insisting that is is still “a problem with the site.”

Furthermore, we have received confirmation from network analysts that the packet loss we have identified at VZW network hops is being caused by ICMP throttling at the network nodes.

ICMP Rate Limiting is a common practice to alleviate network congestion by prioritizing “more important” traffic. How do I explain to Tripawds users that their beloved pets are not important enough for anyone at VZW support to help?

UPDATE: May 10, 2015

Finally. After getting nowhere with Verizon Wireless technical “support” for nearly two months I believe this problem has actually been resolved! At least so far…After I escalated the issue to Executive Management, I was able to work directly with network techs who actually understood me, and identified a problem with packet optimization that wasn’t working properly for everybody. Persistence pays off!

How to Change WordPress Admin Username and Why

I’m reblogging this from our business site for any WordPress Multisite Network Admins looking for an easy way to change their username in the wake of recent brute force login attacks, or wondering what happened to their Super Admin menu items after they did so.

What does this means to Tripawds members? Only that we are on top of things wen it comes to maintaining and securing this community. Thank you for your support.

From the Team Agreda News Blog…

Fair Warning: This is a bit more technical than our usual home based business tips.

With all the news about recent brute force bot attacks on WordPress sites, however, and considering the number of WordPress sites out there, this is vital information for anyone with a WordPress Admin account.

There are plenty of articles about the WordPress Admin Botnet, so I’ll get straight to dealing with it. In short, hackers are breaking into WordPress sites using brute force login attempts on any “Admin” accounts, the default username for site administrators.

The first line of defense is to ensure you have a strong password. Change yours now, I’ll wait. that’s the first thing I did a few days ago as news of the attacks surfaced. As reports increased, more drastic steps were clearly necessary to protect all our websites.

The second (and most effective) step in thwarting these particular attacks is to change your default Admin username. You can’t just do that from your user profile, but there are various methods. One easy way to change the Admin username is to create a new user account with Admin permissions, then delete the original Admin account and transfer all posts to the new user. That’s great for the basic WP install, but it doesn’t help those of us running SimplePress Forums or multisite communities.

NOTE: Deleting an account and transferring the user’s blog posts will not reassign that user’s forum posts, permissions, subscriptions, etc. when running SimplePress.

How to Change WordPress Admin Username via PHP MyAdmin

The following are steps to quickly and easily change the default WordPress “Admin” account username in your database using PHP MyAdmin. This is less daunting than it sounds, but it does assume you have cPanel access and are familiar launching PHP MyAdmin. If not, review these simple steps with screenshots for doing so.

SIDENOTE: What that article does not address, however, is how doing so will affect your Super Admin capabilities if you are a multisite network administrator. I found out the hard way. Read on for details.

1. Log into your cPanel account, launch PHP MyAdmin and open the database for your WordPress installation.

2. Select your wp_users table and edit the row for your Admin account, usually the first, with ID 1.

3. Enter your new desired username in the user_login data field.

4. Click Go. That’s it! Log in with your new username and existing password.

That’s it? Well not quite if you are a multisite Super Admin. Stop here and you will discover the Super Admin menu items have disappeared once you log back in. Don’t panic.

How To Change Default User Name For Network Super Admin Account

WordPress keeps track of Super Admin users in the wp_sitemeta table. Follow a couple more steps to ensure you retain Super Admin powers when changing your Admin username.

1. Change Admin username as described above.

2. Browse your database for the wp_sitemeta table and edit the site_admins row.

3. Note the meta value for existing Super Admins. It will look something like this:

a:1:{i:0;s:5:”admin”;}

In this case, the 5 indicates the username has five characters, and the username is admin. Other variables may be included in this array if you have more than one Super Admin, but you get the point, right?

4. Edit the meta value for site_admins to include the new user name you changed in the first steps above. For example:

a:1:{i:0;s:9:”mynewname”;}

Note that the integer must change in relation to the number of characters in the username.

5. Click Go. That’s it, really!

Follow the simple steps above and log back into your site with your new user name and existing password. If you’re a Super Admin, you will still have your magic menus for doing all your network related stuff. And if you’re a SimplePress Forums Admin, you will still have all your posts and the permissions you need to keep managing your forums. you can even keep your display name as Admin and nobody will be the wiser.

How To Prevent Users from Creating Admin Accounts

If you’re running your own WordPress multisite network, you may want to consider this one extra step to ensure nobody creates another account with the Admin username. Not that they would actually have administrator capabilities, but better safe than sorry…

While logged in as Super Admin visit your Network Settings page and ensure that your list of banned usernames includes “Admin” and your site will never have another Admin user account.

Any questions?

How to Delete All Spam Comments on WordPress Multisite Network

This one is for WordPress mulitisite admins out there looking for a way to bulk delete all spam comments across their entire network and reduce database overhead. Tripawds members can feel free to skip the geekspeak.

Why WordPress stores spam comments in the first place is beyond me, but that’s a different discussion. When I discovered our wp_site_comments table was consuming more than a gigabyte of the database for our Tripawds Blogs multisite network, I was happy to find this gem of an SQL query to delete spam comments and modified it slightly to do so from all blogs at once.

DELETE FROM wp_site_comments WHERE comment_approved = 'spam';

In a matter of seconds I reduced the size of our database by more than a gigabyte!

Yes, WordPress does provide the ability to delete spam comments from the dashboard, but not from all sites on a multisite network at once. This simple query allowed me to delete spam from all sites in one batch. By using phpMyAdmin to execute this SQL query the job is completed quickly, efficiently and without the need to visit each site’s dashboard or the memory overhead created when WordPress attempts to process a large number of comments.

It’s that easy!

Backup your database (Highly Recommended)
Launch PHP MyAmin
Load your network’s database
Click the SQL tab
Run the query above
Optimize the wp_site_comments table to remove any overhead

What does this mean to Tripawds Members?

Nothing really. This is just a reminder of our ongoing efforts to keep this free community online. We spend countless hours managing the Tripawds Blogs network and discussion forums to keep everything operating at top performance. Your continuing support helps make this possible. Thank you.

Tripawds Joins SOPA Strike

On Wednesday, January 18th thousands of web sites will go dark to protest SOPA & PIPA, two US bills racing through Congress that threaten prosperity, online security, and freedom of expression.

To join this effort to spread awareness about this vitally important news, two plugins have been activated across the Tripawds network:

Stop SOPA Ribbon
Help stop American Consorship by putting a ribbon that says Stop SOPA on your WordPress site or network.

Stop SOPA Widget
This plugin adds a widget that will show a modal window to increase awareness about SOPA/PIPA on Jan 18th.

Due to the sensitive nature of this community and the helpful resources all our blogs and discussion forums provide, we will not be shutting down the Tripawds network in protest. These plugins, however, shall remain activated to help us show our solidarity by doing the following.

As of now, all sites throughout the Tripawds network will display the Stop SOPA ribbon in the top right corner.
The Stop SOPA pop-up message shall display on all sites during the scheduled SOPA protest blackout time on January 18th from 8am-8pm EST (1300-0100 UTC). It is easily dismissed and will only be shown to each visitor once.

We shall keep these plugins network-activated through January 24, when congress votes on SOPA and PIPA. The pop-up message will only display during the scheduled strike but you can show it anytime until the 24th by adding #stopsopa to the end of any link to your blog like this:

yourblog.tripawds.com#stopsopa

Please help us put an end to these bills before they become law and end the internet as we know it.

Feel free to comment below, or chime in on this discussion about SOPA. But better yet, contact congress now!