patrick.net

Why the fuck is a water treatment plant controllable through the Internet?

richwicks says

Why the fuck is a water treatment plant controllable through the Internet?

Lol, exactly.

What the fuck were they thinking?

Were they even thinking?

Work from home, man. What could go wrong?

Patrick says

richwicks says

Why the fuck is a water treatment plant controllable through the Internet?

Lol, exactly.

What the fuck were they thinking?

Were they even thinking?

They were thinking, like noobster said, they could work remotely. Also that "Pa$$word" is sufficient, uncrackable protection.

No reason any infrastructure like waterworks, electricity, draw bridges, etc. should be accessible by the Internet. People involved need to get off their lazy asses and travel to site when needed.

Clearly, the hacker used a Dominion voting machine to accomplish this.

HeadSet says

No reason any infrastructure like waterworks, electricity, draw bridges, etc. should be accessible by the Internet. People involved need to get off their lazy asses and travel to site when needed.

I agree, but I also don't depending on your position of what we pay government workers. One employee manning 10 drawbridges on the Chicago River versus having 10 employees at each bridge is a big deal. Given it's probably union, you're looking at $1M just for salary for those bridges versus $120-150k. I don't want to pay for that. Ultimately we have to pay for those 9 workers out of a job if it's automated and remote. It's a fucked up situation we're approaching. I don't know the solution.

WookieMan says

I agree, but I also don't depending on your position of what we pay government workers. One employee manning 10 drawbridges on the Chicago River versus having 10 employees at each bridge is a big deal. Given it's probably union, you're looking at $1M just for salary for those bridges versus $120-150k.

How about $0?

A computer can do this.

Worried about safety? Make 3 independent solutions, and only when 2 out of the 3 agree, do the action. The one that doesn't take the action, debug. If you want to be safer, 3 out of 4, and if there's a draw you just stop and debug.

We can entirely eliminate pharmacists now with expert systems. My niece is a pharmacist, and I expect her job to be entirely eliminated LONG before she retires. Hope she marries a good man and isn't stupid enough to divorce him. Also hope she doesn't become a drug addict, that's a common affliction among them. Bartenders tend to be alcoholics.

WookieMan says

HeadSet says

No reason any infrastructure like waterworks, electricity, draw bridges, etc. should be accessible by the Internet. People involved need to get off their lazy asses and travel to site when needed.

I agree, but I also don't depending on your position of what we pay government workers. One employee manning 10 drawbridges on the Chicago River versus having 10 employees at each bridge is a big deal. Given it's probably union, you're looking at $1M just for salary for those bridges versus $120-150k. I don't want to pay for that. Ultimately we have to pay for those 9 workers out of a job if it's automated and remote. It's a fucked up situation we're approaching. I don't know the solution.

You can have remote access that is not relying on the Internet.

HeadSet says

You can have remote access that is not relying on the Internet.

Internet and phone are the same thing now. The only way to get off the internet for communication is radio transmission.

If you have a DOCSIS cable modem, that's nothing more than a cable television system and the cable television system now that it is digital is delivering packets in basically multicast. It's all internet now.

Basically, your television, phone, any communication is internet at this point.

richwicks says

Bartenders tend to be alcoholics.

I don't find this to be true honestly. Most establishments don't allow drinking on the job and if you want to make the big money you're working until 2-4am. You're not coming to work drunk. Most bartenders I've known will have a night cap, but they can't and won't just sit at work another 2-3 hours pounding booze until 6am. They can't in most cases. Most just want to go to bed after dealing with drunks all night.

This maybe happened in the 80 and 90's, but not now. Half the bars are corporate hell holes. So it's almost impossible to drink on the job. Alcoholics drink all day. Do bartenders drink more than average, probably. The medical definition of an alcoholic is way different than an actual alcoholic. I've witnessed it from multiple family members. Binge drinking is a different animal and maybe what you mean. A true alcoholic cannot function behind the bar in a fast paced environment at an average to decent bar.

richwicks says

We can entirely eliminate pharmacists now with expert systems. My niece is a pharmacist, and I expect her job to be entirely eliminated LONG before she retires.

I've actually gotten wrong prescriptions due to pharmacist error. I'd be willing to try an automated system.

Doctor literally types (no handwriting allowed!) what drug, dosage, etc and then it pops out somewhere. Why should a human be involved at all?

OK, programming errors could fuck it up, but lots of things are like that now.

richwicks says

HeadSet says

You can have remote access that is not relying on the Internet.

Internet and phone are the same thing now. The only way to get off the internet for communication is radio transmission.

Maybe "cellular," which is actually radio transmission. Even though copper is being phased out to homes, we still have a good infrastructure of copper running on poles and underground virtually anywhere. Perhaps this can be repurposed to make point-to-point connections to for remote control of drawbridges, etc. No IP or packets of any kind, just one unique box establishing a connection to another like an old fashioned phone, using protocols unique to themselves.

HeadSet says

Maybe "cellular," which is actually radio transmission. Even though copper is being phased out to homes, we still have a good infrastructure of copper running on poles and underground virtually anywhere. Perhaps this can be repurposed to make point-to-point connections to for remote control of drawbridges, etc.

No. Your cellphone connects to a celltower, and then from there it goes to the internet basically it then goes through that medium and is spit out from another cell tower or (very rarely) a landline.

Internet isn't just IPV4/IPV6 it's ATM packets. ATM is an interesting packet concept. If I recall correctly, it's a 64 bit packet with 2 bytes for addressing. It uses something like onion routing. First you setup the connection points, and then all the connection points know how to connect to the next connection point to get to the destination. Low overhead, low latency. I don't think there is any error correction or detection built into the protocol, that's in the lower layers of the network - and there must be some error correction because I've never heard an error. It would sound like static or at least a short blip.

Everything is digital, and if you are old enough, you know this is a tremendous improvement. My great aunt (and she was great!) used to talk to her family in Poland in the 1970s and 1980s, she had to shout into the telephone to overcome the static. I hate audiophiles because they claim great ability to hear, and swear by analogue, and don't understand the most basic concepts of analogue versus digital. An Engineer is to an Audiophile, as a Doctor is to a Quack.

Anyhow:

HunterTits says

No IP or packets of any kind, just one unique box establishing a connection to another like an old fashioned phone, using protocols unique to themselves.

You can accomplish this with a unique symmetric key over the internet where you have a hardware box that does the translation. The problem is if the box is stolen. Same problem you'd have with a unique phone number. Source phone numbers can be faked.

We CAN solve the problem of security, but the government doesn't want this problem solved. I can eliminate even the possibility a virus exists, but the government crushes it. Do you know we've had secure email since 1994? Yet, nobody uses it. This is PGP. This would be easy to build into an email system today, since it's been in email systems for 25 years - but nobody makes it commercially available..

Most engineers just do a job, few think about why obvious features aren't included. I really can make your computer impossible to hack - from a software point of view - I can't guarantee the hardware. There's backdoors everywhere. The thing is, even if you are targeted, they don't know which backdoor to use. NSA depends on spyware and commonly known backdoors, they are NOT genius mathematicians that can break any code.

I'm sure the NSA can access my hard disk - I have 6 TB - good luck finding the information you're looking for on my internet connection, without me detecting my internet connection is dragging.

We can have complete security, but our government seriously does not want that. 4th amendment died at least 20 years ago.

Patrick says

richwicks says

We can entirely eliminate pharmacists now with expert systems. My niece is a pharmacist, and I expect her job to be entirely eliminated LONG before she retires.

I've actually gotten wrong prescriptions due to pharmacist error. I'd be willing to try an automated system.

Doctor literally types (no handwriting allowed!) what drug, dosage, etc and then it pops out somewhere. Why should a human be involved at all?

OK, programming errors could fuck it up, but lots of things are like that now.

Someone has to fill the bins, if the robot thinks it has drug x, but it is actually z the robot will be wrong every time. But half the job of a pharmacist is watching for and addressing dosing errors, drug interactions etc.. those decisions are harder for a robot to do in a meaningful way. And when a robot does find a problem, can they call the prescribing physician to have them write for the correct dose or medication? Perhaps some day but we are not there yet. Robots on the other hand can and probably should do most sterile compounding (mixing of IV medication) they would be near error proof in that role.

HeadSet says

No reason any infrastructure like waterworks, electricity, draw bridges, etc. should be accessible by the Internet.

Or by the Chinese.

richwicks says

Worried about safety? Make 3 independent solutions, and only when 2 out of the 3 agree, do the action.

Oh boy. I can see myself heading down river, my motor wont start, no wind to fill my sail and I have to get 2 out of 3 computers to agree that they need to raise the bridge and let me through before the current makes me crash into fiberglass splinters.

Of course It's more likely a computer would be on duty than a human who was busy peeing or drunk.

I would throw out an anchor and work on my engine until the bridge was opened.

I'm a chemical engineer who dabbles in IT / programming / controls as a hobby, so this is in my wheelhouse.

These chemical / wastewater/ industrial plants are a unique animal. If somone had ill inent, it would take some specialized knowledge to pull off. Plus most plants have operators in the control room 24-7 keeping an eye on things, as was the case here. A number of things have to go "right" to sucessfully and willfully sabotage a system such as this.

First there is the issue of access. That's the eaiset part in many ways, but by no means "easy". Remote monitoring is more common than it was even 5 years ago, but it's still not commonplace yet. Most systems I've seen are only accessible internally. So you'd have to find a way into the internal network.

Once your "in", that is to say have access to the control panel as if you were a plant operator, you'd then have to know what values to change... And have permission to change them.

That assumes the PLC programmer didn't hard code an upper and lower bound to the setpoint for a particular control variable, which is advisable, since you don't want to give an operator too much control, lest they throw the system out of whack.

Not to mention there would be high and low alarms on most measured values to grab an operators attention if something goes haywire. Rarely can these be overridden by an entry level operator.

So even if one changes the control setpoint from 100 ppm to 1100 ppm, the system would have a high alarm set at say 500 ppm and a high high alarm at say 1,000 ppm to indicate a problem that the operator could respond to.

These alarm setpoints are separate from the control setpoint above and often can hard coded into the PLC program, in which case they can only be adjusted by uploading new code.

OR

if they are able to be overridden at the user level,
you'd need to be logged into the control system as an admin level user and / or need to provide a password before being allowed to change said alarm setpoint up or down. It would be highly unusual for an alarm point to be able to be changed willy nilly by a typical operator.

Setpoint yes (that's what these hackers did) alarm points, no.

Not to mention that if the control scheme is to generally accepted pratices, there would be an interlock between the dosing pump adding the chemical and the pH reading from the pH probe so you don't overshock it too much. (For those of you with pools you know what I mean.)

Sounds like this plant had that, since they said there was some degree of pH control as per the article. With a typical interlock, if the pH gets too high NaOH feed pump would shut down automatically.

Finally, they probably also have a pH balancing step so that if pH ran high (say the chemical feed pump malfunctioned and didn't shut off as expected when the alarm point was reached) they could buffer with an acid to pH balance prior to discharge to the municipal water supply.

(Neutral water has pH of 7; lower than that and your acidic, higher than that and your basic.)

It would take an evil genius with (or a small team of experts) to sabotage a plant, even a small municipal one.

Good PLC programmers are generally just that... good programmers who have a minimal understanding of process. Process Engineering experts (such as myself) are the guys (or gals) telling them what to program and how the control scheme for a particular unit operation should work... but few of us process geeks know the intricacies of DSC control systems and / or PLC programming.

Mabey this was a warning shot, possibly was an inside job, else it was some dumb kids messing around who did no real damage. Most modern plants have enough redundancy to protect against something like this, which could have easily have just as easily occurred by operator error as an outside hacker. (For instance attempting to increasing the setpoint from 100 to 110 and adding and having fat fingers and completely entering a bogus value of 11,100.)

One final thought... Thankfully most sensors measuring key process variables at the local level are still hard wired, and most have a local LCD display on the head of the transmitter. (The wireless sensors are prone to interfernce and are not practical for most facilities.) Hard wired sensors give added redundancy. If the control room shows a pH of 10 when your expecting 7.2, you can radio to a field operator to look at the instrument and confirm (or deny) the reading on the instrument.

Most systems are SIL2.. that is to say they have at least two separate failsafe before it all hits the fan.

If any of you ever run into this in your line of work and need my assistance, I do consult on matters such as these.

Holy shit NYCdude. Very thorough explanation. Thanks, good stuff.

seems like inside job such as chicom hackers within these companies or an insider was paid off by chicoms to provide backdoor info to the computer systems... look at how chicoms got the security clearance database files from the US Office of Personnel Management ... and I just read also about a navy chief and fire control specialist being arrested for espionage for handling classified info to a foreign agent...

https://www.usatoday.com/story/tech/2024/02/22/att-outage-service-down/72695579007/

https://www.startribune.com/cyberattack-hits-unitedhealth-group-unit-by-actor-with-suspected-nation-state-ties/600345420/

https://nypost.com/2024/02/22/lifestyle/pharmacies-nationwide-face-delays-as-health-care-tech-company-reports-cyberattack/

.

wonder if this is another Chicom hack

https://www.nbcnews.com/tech/tech-news/multistate-911-outage-shows-fragility-systems-experts-say-rcna148475

.