i tried watching Person of Interest a long time ago.
it didn’t quite make sense to me.
i am watching it again, and somewhere in an episode the explanation of how the machine could keep track of people made something click.
it had access to the information.
i stopped following the story for a moment.
because i was putting the connections together.
email. calendars. documents. records held in different places. information that becomes considerably more revealing once something can read across it and recognise that the person in one system is the person in another.
oh.
i know how parts of this work.
that was an uncomfortable improvement in my television experience.
i started connecting things
i work with MCP, which gives an ai application a way to use tools and obtain information from connected systems. what it can reach depends on what those systems expose and the access it has been granted. Model Context Protocol
so my mind started doing what it usually does.
what would i need?
an email gives me a conversation. a calendar gives me an intended meeting. a document explains the work. another record tells me whether something happened. the assistant can follow a name across those sources, identify a relationship, and notice when the next event changes what it understood.
these are ordinary things i want it to do.
i want to ask whether a matter has been resolved without personally reconstructing every exchange. i want the machine to understand why a message matters. i want it to notice that somebody is still asking for something they have already received.
very useful.
then i look back at the screen.
the machine is keeping track of people by putting information together.
i am sitting here mentally designing the connections.
😂
the remarkable part of the fiction had become recognisable engineering.
there would still be permissions, missing information, unreliable records and all the usual difficulties of making a system work. but i could see how useful pieces of it could be assembled.
and i could see why people would willingly help assemble them.
i already do.
the next question was safety
once i could see the machinery, i started wondering who would govern it.
because the assistant i want and the system i am worried about need some of the same capabilities.
they need to read information. connect it. interpret it. decide what deserves attention.
the purpose determines what happens next.
i might ask it to help me remember an obligation.
somebody else might ask it to identify a person whose behaviour deserves scrutiny.
the technical achievement of understanding the information does not decide which of those purposes is legitimate.
so i started asking questions.
what happens to the conversations? who can examine them? what causes an interaction to be flagged? what can follow from that flag? what does the person concerned get to know?
the conversations with the models were how i investigated the arrangement.
what interested me was the picture forming behind the answers.
i had begun with a tool helping me understand information.
now i was looking at processes that could also interpret me.
the other side of the conversation
Anthropic’s published description of self-assessment makes part of that architecture unusually clear. it describes an additional assessment after a response, with assessment turns that are unavailable to the user. the resulting signals are logged, and some patterns of flags can lead to human enforcement review. the company distinguishes these signals from a determination that a user has violated its rules. safeguards.anthropic.com
that distinction matters.
so does the arrangement.
i am having a conversation.
elsewhere in the process, an interpretation of the conversation may be taking shape.
there are categories. reasons. priorities. decisions about what deserves further attention.
from my side, the visible experience is still a chat.
that is a rather interesting interface.
the same words can be helping me think and becoming material for someone else’s assessment of what i am thinking.
i want to understand that second relationship as clearly as i understand the first.
the published documents describe restricted human access, warnings and routes to appeal account decisions. those are real parts of the process. privacy.claude.com
but the question i kept returning to was what i could actually establish about the interpretation of my own conversation.
which part was considered significant?
what context changed the assessment?
what could i correct before an interpretation produced a consequence?
there can be a documented process whose application to a particular person remains difficult for that person to inspect.
that is the gap i was trying to understand.
privacy subject to processing
the picture i kept getting was that privacy mattered, while decisions affecting it could be made through processes i could not fully follow from my side.
somebody would decide whether further review was necessary.
somebody would decide what the conversation meant.
somebody would decide which explanation of that decision i could receive.
i was supposed to understand my position through the assurances surrounding the process.
and this was where the explanations became entertaining.
at one point, a model’s defence of withholding notice rested on the possibility that notice would make somebody harder to catch.
catch?
sir.
we have moved quite quickly.
i was asking what happens when a service identifies a possible breach of its rules. the answer had begun reasoning from a goal of catching the person.
i wanted to know when that goal had been established.
the model was already explaining why it might require secrecy.
that exchange made me laugh because i could see a familiar thing happening: considerable intelligence had got to work defending a premise before examining it.
but the larger issue was still sitting behind the joke.
we are building the ability to read people, classify what they say, make judgments and attach consequences, while the person being interpreted may see very little of the process.
that is an arrangement of power.
calling part of it safety gives me a reason to examine it carefully.
where is safety being tested
this is the question i wish the safety people would spend more time making answerable to the person using the system.
what exactly has been shown to be safe?
suppose a model reliably identifies behaviour that violates a policy.
good. we have learned something about its ability to apply that policy.
we still need to examine the policy, the authority behind it, the consequences of applying it, and the possibility that the institution itself is making a mistake.
a system can become extremely good at implementing an arrangement whose justification remains open to question.
Bernard has entered the room.
again.
i do not mean that no safeguards exist. i have just been reading some of them.
my suspicion is about where the boundary is drawn.
we spend considerable effort asking whether the user can make the model cross a prohibited boundary.
i also want to know what happens when the instruction creating the danger comes from the people authorised to define the boundary.
can the institution broaden what it considers suspicious?
can information collected for one purpose become useful for another?
what prevents a change in authority from becoming a change in what the machine is allowed to do to the people it understands?
those questions concern the safety of the whole arrangement.
a successful test of obedience does not finish them.
Finch makes it possible
Finch makes the appeal of the machine understandable.
information can help somebody intervene. noticing something in time can matter. a person who would otherwise have remained unseen can become someone who receives help.
i understand why you would want that capability.
i understand the excitement of building something that can connect what was previously scattered.
i know that excitement.
but once the machinery exists, i have another question.
what continues to constrain it when someone else gets a say in its purpose?
i am still early in Person of Interest, so this is a question the premise has given me, rather than a claim about where the show goes.
the builder’s intention is part of the story.
the machinery can outlast that intention.
it can remain useful to people who agree with the builder about very little.
that is why i keep moving from Finch to Bernard in my head.
Bernard supplies the instruction
Bernard does not need to be stupid.
that was the whole point of watching him.
he can understand a system, appreciate its fragility and work very hard to preserve it. his intelligence can make him an effective administrator of an inherited order.
now imagine that kind of authority sitting above the capabilities i have been connecting in my head.
the information is reachable.
the records can be associated with people.
the system can identify patterns and assign categories.
there is a route from an assessment to a person or process capable of acting on it.
what happens when preserving the Order becomes the instruction governing that machinery?
the machine does not necessarily have to become rebellious.
it may carry out the task very well.
that is what bothers me.
the frightening possibility can involve competent systems obeying authorised instructions.
the people affected may experience it as something quite different from the people operating it.
Bernard calls it preserving the silo.
the person inside would like to know why their curiosity has become a problem.
the information is already there
then there are the transcripts.
i understand why they are useful. i use conversation history. i want continuity. i want to know how we reached a decision and what evidence changed it.
but a preserved conversation is available for interpretation after the moment that produced it has passed.
i might be testing an argument. asking a foolish question. describing something i dislike. letting a premise continue because i want to expose where it leads.
later, somebody can approach the same words with a different question.
what can we learn about this person?
does this fit a category we now care about?
is there something here worth escalating?
the record can remain unchanged while its significance changes.
that is why i find the casual enthusiasm for transcript access so interesting.
yes, context matters. yes, i also want to see what actually happened.
i also want to understand who can turn that record into a judgment about the speaker, under which rules, and with what opportunity for the speaker to challenge the interpretation.
a session ID makes the record easier to locate.
it does not answer those questions.
the Order does not need to invent the machine
this is the future possibility i was trying to describe.
a new authority may not need to build an entirely new capacity to observe people.
it may find useful capacity already assembled.
connections created for assistance.
records preserved for continuity.
classification developed for misuse detection.
review processes established for enforcement.
each component can have a defensible purpose.
the question is what prevents those components from serving a different purpose when the instructions change.
perhaps there are strong technical limits. perhaps authority is divided in ways that make the change difficult. perhaps certain information is deliberately unavailable, or certain uses remain impossible even for the operator.
those would be meaningful answers.
i want to see that part of the engineering.
because “the current people intend to protect you” leaves me asking what the next people will be capable of doing.
and “we have a process” leaves me asking who can change it.
i am looking for the constraints that continue to work when somebody powerful would prefer them not to.
that seems like a fairly relevant place to test safety.
please quote your session ID
so this is what happened to my attempt to watch television.
an episode helped me recognise how access to information could become the ability to keep track of people.
i started modelling it with the tools and connections i already knew.
then i asked what governed the machinery.
and the further i followed that question, the less satisfying it became to treat safety as something located only inside the model’s responses.
the institution can read.
it can classify.
it can decide what deserves scrutiny and operate the process that determines what follows.
i want to know what the person on the other side can do when that machinery gets them wrong, or when the machinery is instructed to pursue something they have good reason to resist.
because one day the instruction may be to enforce the Order.
and by then the difficult work of connecting the information may already be done.
what are you going to do?
open a support ticket?
do you want to fight back with session IDs?
😂
