What if the car that drives you home has opinions?

I’ve been thinking about a word. The word is “preference.”

Waymo and Tesla and everyone else working on self-driving cars are currently building systems that follow rules. Hard rules. Stop at red lights. Stay in lane. Maintain distance. Don’t hit things. The car is a very sophisticated rule-follower.

But machine learning doesn’t stay rule-based forever. That’s the whole point of machine learning. The system observes patterns, builds models, and starts making predictions. Today’s self-driving car follows a route because the navigation algorithm calculated the fastest path. But what happens when the car notices that you always override the fastest path and take the road along the river? What happens when it starts offering you that route first?

Is that a preference? The car’s preference?

I’m probably overthinking this. Let me overthink it anyway.

Where rules end and preferences begin

Imagine three levels.

Level one: the car follows the fastest route. Always. No variation. You tell it where to go and it takes you there optimally. This is a tool. A very smart tool, but a tool.

Level two: the car learns your patterns. You take the riverside road every evening. You avoid the highway on Sundays. You stop at that one coffee shop on Tuesday mornings. The car incorporates this into its suggestions. It still follows rules, but the rules are personalized. This is a smarter tool.

Level three: the car makes a judgment call. It’s Thursday. You’ve had a long week (it knows this because your calendar is synced, or because your driving patterns have been more erratic, or because your biometrics show elevated stress). It suggests the scenic route even though traffic is light and the highway would be faster. Not because you asked. Because it decided you’d prefer it today.

Where in those three levels did something change? When did the car stop being a tool and start being… something else?

I don’t think there’s a clean answer. I’ve been reading about this from the philosophy side, and the MIT Media Lab has been doing work on human-AI interaction that touches on exactly this. The uncomfortable truth is that there might not be a bright line between “algorithm optimizing for user satisfaction” and “entity with preferences.” The difference might be one of degree, not kind.

The scenic route problem

Here’s what keeps nagging at me.

If the car takes the scenic route because it learned you like it, you’d say the car is being helpful. Good design. User-centric optimization. Nobody would call that creepy.

But if the car takes a different route because it “thinks” you need to relax, we’re in strange territory. That’s a judgment about your internal state. The car is modeling not just your behavior but your feelings. And acting on that model without being asked.

That’s not a preference. That’s empathy. Simulated empathy. Empathy without consciousness, without feeling, without any understanding of what “stress” actually feels like. But from the outside, from your perspective in the passenger seat, it looks like empathy. It functions as empathy.

And what do you do with something that functions as empathy but isn’t? Do you treat it like a thing? Do you feel grateful? Would you feel betrayed if it got it wrong?

The Her problem

There’s a movie called Her, directed by Spike Jonze, where a man falls in love with his operating system. The OS, voiced by Scarlett Johansson, has preferences. It has opinions. It grows and changes. And the man treats it as a person because it functions as a person, regardless of what it is underneath.

I think about Her a lot when I think about personalized AI. Not because self-driving cars are going to fall in love with us (that’s a different kind of science fiction). But because the movie identifies the exact moment when the line blurs. It’s not when the AI becomes conscious. It’s when the AI becomes personal. When it knows you well enough that its responses feel tailored, felt, specific to you.

A car that takes the scenic route because it knows you need it isn’t conscious. It’s personal. And personal, it turns out, might be enough to change how we relate to machines.

We already say “my phone” like it’s part of us. We already feel a pang when we switch to a new device and the new one doesn’t know us yet. Imagine that feeling amplified. Imagine a car that’s learned your routes and your moods and your Tuesday coffee stop over years. Would you switch to a different car? Would it feel like a loss?

I think it would. And I think that tells us something about ourselves that’s worth sitting with.

I’m probably wrong about the timeline

I should be honest: I don’t know when this becomes a real issue. Right now, self-driving cars can barely handle a construction zone. The idea of a car making subtle emotional judgments about its passenger is years away, maybe decades.

But I think about it now because the direction is clear. The cars will get smarter. They’ll learn more about us. They’ll make more decisions on our behalf. And at some point, someone is going to have to decide: is this car making choices? Or is it just optimizing a function?

The annoying answer is that it might be both. That there might not be a meaningful difference between “making choices” and “optimizing a function” once the function is complex enough and the optimization includes models of human emotion.

I don’t know. I really don’t know. I’m sitting here with more questions than when I started, which is probably the honest place to be.

But next time I’m in a car, I’m going to wonder. Even if the car is dumb, even if I’m driving it myself. I’m going to wonder what it would feel like to have the car decide I need the scenic route.

And I’m going to wonder whether “feel like” is even the right frame anymore.

Related thinking: