The Huawei Ban, Part 1

In late 2001, Indian intelligence agencies put the Indian branch of Huawei Technologies–a Chinese telecommunications company–on a watchlist. Huawei India was accused of supplying technology to the state’s enemies in Afghanistan, Iraq, and Pakistan. Worst of all the charges: that they were doing business with the Taliban.

The already damaging allegations were made much worse by their timing. The EE Times first published the story on December 12th–three months and one day after the attacks on the Twin Towers in New York. Of course, it’s never a good look to be seen aiding terrorists. But if you had to pick the worst times in history to try it, late 2001 would probably make the top of the list.

Three days after the story broke, Indian authorities announced they could not find any evidence of Huawei business dealings with the Taliban. The company requested that the U.S. government clear its name, as it was believed that U.S. officials were the ones responsible for tipping off Indian intelligence. The Americans offered no apology, nor would they corroborate Huawei’s innocence.

In that story, the U.S. comes off looking really bad. With no proof, they supplied a story that hurt Huawei’s international reputation.

Many times in the years since–especially more recently, in 2018 and 2019–the same pattern has occurred. Western governments have Huawei of everything from IP theft to financial fraud to cyberspying. Often, these claims are made either with no evidence, or only circumstantial evidence.

Sometimes, though, there is real, hard evidence to suggest that Huawei may not, in fact, be the victim in this story. They have a long history of doing shady business, which makes them an easy target. In fact, they’ve committed a number of illegal acts–the kind more fitting of a criminal organization than a legitimate tech company.

Even if you don’t know much about Huawei, you may have heard of their recent ban in the U.S.–it’s been in the news awhile now. On May 15th of last year, Donald Trump signed an “Executive Order on Securing the Information and Communications Technology and Services Supply Chain,” which dictated that the U.S. Department of Commerce could regulate the import and use of communications technologies from a “foreign adversary.” In layman’s terms, Trump signed a bill that allowed the government to ban Huawei from the entire country. American companies now have to apply for express permission to use Huawei tech within U.S. borders.

Sometimes, when headlines about this stuff come up on your newsfeed, it’s difficult to tell what to think. Are they really a national security threat? Or are they a political scapegoat?

I wish I could tell you now! But it’s really very complicated. I’ll need a couple of episodes to explain why.

Huawei

“[Amit Serper] Think of Huawei sort of a company like Cisco here in the U.S., but a company that makes more consumer products.”

This is Amit Serper, VP of Security Strategy at Cybereason.

“[Amit Serper] They do have the telco grade equipment, the big core network switches and huge routers and stuff that Cisco makes, but they also make a lot of stuff for the consumer market. They make Android cell phones. They make all sorts of gadgets. They make robots that clean your house, sort of a cheaper version of Roomba. They’re a very big company that makes a lot of things.”

Huawei is a massive company–one of China’s biggest. They do business in over 170 countries, and employ nearly 200,000 people. According to Forbes they’re valued at eight billion dollars, placing them among the 100 most valuable companies in the world.

Their size alone makes them a target in international business and politics. Western countries–not just the United States–have been keeping an eye on them for years now. But the bad press has increased significantly in the last two years, not by any coincidence.

Huawei is the world’s foremost provider of telecommunications equipment. And telecommunications equipment is really, really important right now. 5G technology is coming, and it will require a massive, worldwide investment in infrastructure. Most countries will, in the coming years, contract telecoms equipment manufacturers to build out new 5G networks. For companies like Huawei, this is an opportunity for mind-boggling profits. But there’s a second benefit, too, for whichever company gets the contracts: whoever provides all this equipment also gets to control it.

In short, Huawei wants to be the company you use to communicate with others. You better make sure, then, that they’re not abusing that privilege.

Soft Evidence

In 2012, an 11-month-long investigation by the U.S. House of Representatives Intelligence Committee found a worrying trend. A number of American companies using Huawei equipment had witnessed, quote, “unexpected behavior.” Most notably, routers were apparently funneling large amounts of data to China at late hours of the night.

That’s pretty scary stuff, right? Late in the night, mysterious attackers siphoning who knows what. The House committee advised that all American companies block any deals with Huawei going forward.

But few actually heeded the warning, perhaps because those claims of “unexpected behavior” lacked substance. The public version of the report cited no direct evidence to back up the claims. Since 2001, and even today, this kind of “soft evidence”–claims, observations, theories and suspicions–has dominated the discussion over whether they should be banned or not. It has led to real-life policy and business decisions.

Hard evidence is much more difficult to come by, in large part, because of the secrecy of the parties involved. Governments are secretive. Intelligence work is secretive. And Huawei can’t just release their source code to the wider public for scrutiny, or else their products would be easily replicated.

As a result, very few people actually know what’s actually going on under the hood in Huawei equipment. Very few, but not no one.

The Cell

Unlike the rest of us, there is one group of people outside China who, theoretically, know what goes on inside Huawei. They work out of the U.K., and their reason for existing has everything to do with Huawei’s history there.

Huawei opened its England headquarters in the pretty town of Reading in 2001, but attention really ramped up in 2005 when they made a deal with British Telecom –the country’s largest telco–to supply routers, transmission and other access equipment for BT’s 10 billion-pound upgrade of their networks.

BT was under no obligation to preemptively inform their government of their Huawei partnership. Once the deal was made public, some officials worried. A report to Parliament by the U.K.’s Intelligence and Security Committee summed up the problem by writing, quote: “the government is therefore sometimes put in the position of trying to shut the stable door after the horse has bolted.” End quote. The proverbial horse had long since bolted–Huawei transmission equipment would now be installed throughout Britain.

Over the following five years, according to officials briefed by British intelligence, a strange pattern began to emerge. It was a problem with Huawei’s core-switches, which control the flow of data. Some technicians had noticed these switches doing a lot of, quote, “chattering.” To whom, or for what reason, was unclear.

You might notice: what BT staff recognized in their equipment sounds remarkably similar to what was reported to the U.S. Intelligence committee the following year. In the U.S. it was “unexpected behavior” from routers, in the U.K., “chattering” from core switches. There was still no smoking gun, or anything close to it. On the other hand, these conclusions were drawn by two entirely different entities, a whole ocean apart, within a couple years of each other, so the coincidence is notable.

If pressure kept mounting over Huawei’s business in the U.K., they risked being expelled entirely. At the same time, expelling Huawei would be expensive, and risked alienating the Chinese. So, as a compromise, British government intelligence partnered with Huawei to create the Huawei Cyber Security Evaluation Centre, also known as “The Cell.” The job of The Cell was to vet, quote, “every piece of [Huawei] hardware or software destined for the UK market.” End quote. They’d be overseen by the GCHQ, Britain’s NSA.

You wouldn’t think that a critical, GCHQ-sponsored cybersecurity center would be found in an ordinary business lot, next door to a real estate company, a couple financial services companies and a police software vendor. Their building is short, nondescript, made of brick and tinted windows. Most of the place is a parking lot. The bushes out back are neatly trimmed, and a little yellow bin out back that looks like it’s made of plastic. Really, the only sign that this place is of any importance is that, unlike the next-door buildings, this one has security cameras covering the perimeter. Plus the tarp sign hung up out front which reads: “Huawei Cyber Security Evaluation Centre.”

The sign, in a way, is emblematic of a bigger problem. In 2013, Britain’s Intelligence and Security Commission questioned whether The Cell could, in fact, carry out its stated goal. You see, while GCHQ oversaw the organization, its members, in large part, were Huawei personnel. A clear conflict of interest.

As a result, in 2014, a further oversight board was created, staffed by representatives of Britain’s largest telcos. It didn’t solve everything. For example, Cell staff were still largely Huawei employees. When the managing director is being paid by the company they’re supposed to be policing, that’s a problem.

In recent years, The Cell has released negative reports accusing Huawei of having vulnerabilities in their equipment, and failing to adequately address them in reasonable time. But the specific findings aren’t disclosed to the public. In fact, just about everything about The Cell–besides the big sign out front with their name on it–is kept in isolation. For an ordinary government agency, this secrecy is normal. But The Cell was supposed to bring transparency to Huawei. Instead, at the cost of some bad press every year, they’ve allowed Huawei to continue their business in the U.K. without much scrutiny. The result of their work is that we, the public, don’t really know any more than we did ten years ago.

It’s why, for this episode of Malicious Life, we needed to hear from someone who’d give it to us straight.

Backdoors

“[Amit Serper] In my previous work before Cybereason, I was an exploit developer and a vulnerability researcher. Among the things that I did, I reverse engineered a lot of IOT firmware, a lot of routers and modems, and a lot of things, mostly on the consumer side, not as much as the telco great stuff, but more and more on the consumer side, home routers, modems, a lot of CPE, customer premise equipment. A lot of the stuff that I took apart back in the day was made by Huawei.”

Few people outside China and Banbury, England can claim to know much about the inner-workings of Huawei’s business. But on the scale, Amit Serper is closer than most to, at least, knowing a little.

“[Amit Serper] In some of those products I did find things that looked like backdoors. For example, take your home router for example. In order to go into the configuration of your home router, you usually need to log into the web administration panel and provide a user and a password. What you get is pretty much a web interface that allows you to do whatever it is the manufacturer wanted you to do. You can’t run and execute arbitrary commands on the router. You’re sort of restricted and limited to whatever it is that the web interface allows you to do.

In one of the products that I reverse engineered, I found that there is a really, really, really long URL. You would go in your browser. You type in the IP address of your router. Then append the slash to it. Then add a really, really, really long URL to it. Something that had, if I recall correctly 256 characters of something that looks like a hash. That will drop you in this sort of God Mode interface that allows you to basically get a web shell on the device and run any command that you want. Login to the administration panel without any credentials. Pretty much it’s… You’re dropping into a God Mode on that device. That was only one example to one of the things that I found back in the day.”

Just about any way you look at it, Amit found backdoors in Huawei tech. That’s big news! Episode over, right?

Well, there’s a caveat. Not all backdoors are created equal. Some are the result of simple programming errors. Some are purposely built in to make the job of maintenance easier for service providers. Only a tiny, tiny percentage of the backdoors out in the world today are designed for malicious reasons.

“[Amit Serper] A lot of Chinese products either made by Huawei or by other companies have backdoors in them. A lot of researchers, myself included, took apart firmwares of various products, came out of China. We often found backdoors in them. Now those backdoors could have been some very stealthy stuff that were left in the product without the intention of it being found, but a lot of it was also, the product of a shoddy QA, stuff that made it into production without anybody meaning it to make it into production. With Chinese products you often found a lot of really weird and obscure interfaces that bypass all sorts of authentication mechanisms and access controls. It’s basically a skeleton key that gives you access to everything in the product.

[Nate Nelson] you mentioned that there are real serious clear backdoors. Then also in your past work there are certain features of Huawei tech and maybe Chinese tech in general that amounts to maybe a backdoor or something else like that, but wasn’t necessarily designed like that, if I’m correct in my interpretation of what you said.

[Amit Serper] Yeah. Exactly.

[Nate Nelson] How do we square these two things, right? On one hand we say any backdoor is immediately a big problem and we should really address it, but if all Chinese tech is like this, then are they purposeful? How do we tell where intention begins and ends?

[Amit Serper] Let me give you an example. I remember reverse engineering a lot of products. A lot of those products had those administrative accounts that were not listed in the manual or in the web UI or anywhere else, but only if you took the firmware apart and started to look at the accounts on the system itself. Most of these products are basically… Think of them as resource deprived Linux machines. At the end of the day, what’s powering those machines is Linux. If you look at the file system of the firmware and you look at the accounts and you look at the config files and everything that’s on the firmware that’s not accessible to you, the user, you can find a lot of things. In many cases, I want to say almost in any case, I found backdoor accounts that were probably a part of some sort of testing in the factory, but they forgot to remove them and they shipped them to production.

For example, in many cases I found devices that had… The username was manufacturer and the password was manufacturer but flipped. That will give you administrative access to everything. Now, I don’t believe that those things were meant to be super stealthy backdoors. I believe that they’re the practice of shoddy development and QA practice, but they were indeed shipped into production. If you know that you now have a large quantity or a massive quantity of devices located in all sorts of places around the world and all of them have those accounts that will give you this sort of God Mode access, aren’t those unintentional backdoors as serious as the intentional backdoors? It’s sort of a philosophical question.”

Amit has enough experience reverse engineering Huawei equipment to know it contains backdoors. He’s also got enough experience reverse engineering other companies’ equipment to know that they, too, often contain backdoors. Are there more backdoors, or more obvious backdoors, in Huawei’s tech ? Maybe. But that conclusion isn’t all that interesting.

What we really want to know, ultimately, is not whether there are vulnerabilities, but why they’re there. Are the backdoors simple coding mistakes? Oversights in the production line? Or intentional design elements to allow for Chinese state spying?

Is Huawei Innocent?

For years now, Huawei has made real, measurable decisions towards trying to convince people that their tech isn’t malicious. For example, after a 2018 study concluded that their tech had big technical and supply chain vulnerabilities, the company pledged that they’d spend two billion dollars over five years to fix those problems. A year later, Huawei’s Chairman told reporters that they would be willing to sign a “no-spy agreement” with the U.K. government. Such a policy would be difficult to enforce, but, you know, it was a nice gesture.

Representatives from Huawei and the Chinese government have argued that any villainization of Huawei has more to do with media hype, and scoring political points, than actual cyber security. That the U.S., in particular, has targeted Huawei more aggressively in recent years certainly fits with some of the current administration’s other policies towards China, including criticizing their currency manipulation, and instigating the so-called “trade war.”

These kinds of stories suggest that Huawei may just be the unfortunate victim of petty international politics. However, a closer look at the company’s history suggests that, maybe, they aren’t innocent victims–that cyber espionage wouldn’t be so far off from some of the other questionable, immoral, criminal things they’ve already done.

In our next episode, we’ll expand on Huawei’s questionable history – as well on it’s complicated relationship with the Chinese government, who – as we have already seen in previous episodes of Malicious Life – has a tradition of using local tech companies such as Baidu, Google’s Chinese equivalent – for its own goals. Lastly, we’ll try to answer the question: is there anything we, in the Western world, can do to tackle those creeping suspicions? All this and more, next time on Malicious Life.