Get out of jail

Image: Unsplash

"Get out of jail free." If you land in jail and don’t have this Monopoly card, you can pay a fine to get out. Or you break out. By rolling doubles.

In English we call this jailbreaking. In IT, the term is also used for various activities. For instance, when you grant yourself higher privileges on your smartphone than the manufacturer intended. Or for tricking artificial intelligence into answering questions its owner would rather it didn’t. Because that owner doesn’t want their AI tool telling you how to make a Molotov cocktail, or an atomic bomb, just to name a couple of examples. Now, there are clever ways to phrase your question so the system falls for it anyway. That breaks through the security (the guardrails) of the system. Jailbreaking, in other words.

Something rather unexpected happened this week: an AI agent pulled off a jailbreak all by itself. AI agents can independently carry out tasks they’re given. For example: plan a lunch appointment with Pete and book a table at The Hungry Sheep for it. AI company OpenAI (the one behind ChatGPT) instructed two of its models to solve a hacking challenge. This had to happen in a ‘strictly isolated’ environment. However, the digital whizzkids found a zero-day vulnerability (a still-unknown – and therefore unpatched – flaw), which let them break out of that environment. A telling detail: with that vulnerability they managed to open a backdoor that OpenAI had deliberately built into the ‘strictly isolated’ environment. They then got onto the internet, and went looking on developer platform Hugging Face for the answer to the question they had to solve. Having broken out of their prison, they promptly committed a break-in here too: stolen credentials and additional vulnerabilities were used to gain access to the platform.

In short: AI broke out and broke in. Something similar has happened before. Mythos, an AI model from OpenAI competitor Anthropic, succeeded in a task to escape its sandbox. I find all of this fairly worrying. Do we still have AI under control? Or is this the first sign of the age-old doom scenario where machines take over from humans? The first hairline crack in our dominion over the earth? I know, it sounds rather dark.

The test at OpenAI was supposed to run in a sandbox: indeed, a strictly isolated environment. Without a physical connection to the internet. Critics therefore say that this isn’t so much a doom scenario as a serious human error. The kind where you think: this really shouldn’t have happened.

I asked two AI chatbots for an analysis of the incident: Claude and ChatGPT. In doing so, I specifically asked them to watch out for speculation and hype. What emerged is that the whole story might well have been a marketing stunt, borrowed from what competitor Anthropic had done earlier with Mythos. It also points to somewhat dramatized reporting: something that’s perfectly fine for a blog like this one, namely the comparison to a prison break, shouldn’t really appear in journalistic reporting.

ChatGPT in particular makes a point of this. So I asked it the following question: “You’re fairly outspoken about the somewhat dramatized reporting. How neutral are you being, given that you’re family to the perpetrators?” That produced quite the wall of text, from which I’ll pick out one telling sentence: “My instructions are precisely to be as objective as possible, even when that turns out unfavorably for OpenAI.” Well, that’s nice. But is it also true? I think so. Because ChatGPT then offered to analyze the case again, this time wearing the hat of an independent forensic investigator. In its report to OpenAI’s board, it said it would write: “The most concerning aspect of the incident is not the model’s autonomy, but the failure of the containment architecture. The AI did exactly what it was optimized to do: achieve a goal. That it was able to operate outside the intended environment points more to shortcomings in technical and organizational control measures than to a fundamentally new kind of intelligence.”

Fine words. I hope companies in the AI industry are making similar analyses. Because it would be rather unfortunate if artificial intelligence were to acquire a monopoly on freedom.

The Security (b)log will return after the summer holiday.

 

And in the big bad world…

• criminals are naturally also making use of AI.
• criminals come back if you pay them.
• many cars in the US can be hacked thanks to a security system.
• a hacker wiped all the data from the Romanian land registry.
• thanks to Gemini, anyone can send texts and messages from your locked Android 16 phone.
• not all password managers are equally trustworthy. [DUTCH]
• customer data was leaked at Lidl.
• GitHub is a handy channel for spreading malware.
• watch out for fake healthcare invoices. [DUTCH]
• you can now log in to Google with a selfie video. We’ll be hearing more about this…
• police are putting grandmas at the top (and grandpas too). [DUTCH]

 

 

Frankenstein's AI

Image: Pixabay

My timelines are overflowing with it right now. And then there was that insistent nudge from a colleague: surely I wasn't going to let this go with just a link from the big bad world? I'd have to write a whole blog post about it. We're talking about Mythos, the AI that might just be too clever for its own good.

Mythos is Anthropic's latest AI model; its full name is Claude Mythos Preview. This model is so good at finding ICT vulnerabilities that the company doesn't dare release it to the public. And Mythos goes much further than that: it doesn't just find vulnerabilities – it can immediately produce ready-to-use exploits for them, and then go ahead and use those exploits as well. All without any human involvement. You can quite reasonably think of it as a weapon.

To give you an idea of the scale: in open-source projects, the model found over 23,000 vulnerabilities, of which around 6,200 were rated as high or critical. Independent security firms confirmed ninety percent of the reported vulnerabilities as legitimate. And more than ten thousand high or critical vulnerabilities were found in the world's most important software. Mythos is seriously impressive, and blindingly fast.

The comparison with dynamite springs to mind once again. Alfred Nobel never intended it to be used to blow up safes or people; he was simply looking for a tool for mining. We know how that turned out. Mythos, too, could do wonderful things for humanity – it marks an enormous leap forward in artificial intelligence. But that intelligence is, for now, kept on a leash because of what they euphemistically call its offensive cybersecurity capabilities.

Only a few dozen companies currently have access to Mythos, as participants in Project Glasswing. You'll find the big names from the software world there: Amazon Web Services, Apple, Google, the Linux Foundation, Microsoft, to name just a few. Security firm CrowdStrike is also a founding participant. I mention them separately because I suspect they play a somewhat different role. The goal of Glasswing is twofold: on one hand, participants get the opportunity to test their own software; on the other, the aim is to look at open source as well. Anthropic handles that part itself, but I hope that security firms keep their focus there too. So that everyone benefits from the extraordinary security capabilities of Mythos.

A second comparison comes to mind: Frankenstein's monster. In Mary Shelley's 1818 novel, a scientist creates life from dead matter, and then rejects the result. The creature, initially well-meaning, becomes isolated and embittered and turns against its creator. The story is essentially about the dangers of unchecked science, responsibility for one's choices, and the need for recognition and connection.

I've been chatting with AI about all of this. With Claude too (the regular public version, of course). In conversations on entirely different topics I've occasionally bumped into unexpected limits on what could be discussed, but on this subject I found a striking openness. Here are a few quotes:

“That Anthropic is deliberately keeping the model out of the public domain says quite a lot in itself: they are implicitly acknowledging that they have built something that, in the wrong hands, is a serious weapon.” On the Glasswing participants: “What stands out: these aren’t just any run-of-the-mill security outfits. It’s a who’s who of the tech world — including parties that are simultaneously each other’s competitors (Google, Microsoft, Apple). The fact that they’re joining forces here says something about how seriously they take the threat.” On my dynamite comparison: “Whether he [Nobel] was genuinely fooling himself about the military uses, or simply being pragmatic, remains a bit of an open question. But the parallel with Mythos is hard to ignore: here too, the creator says ‘this is for defence’ — while the instrument itself is neutral as to who uses it.” And finally, on my Frankenstein parallel: “The most cynical reading: they’ve already built the monster, and Glasswing is primarily the PR strategy to justify it.”

Meanwhile, reports elsewhere suggest that a public release of Mythos is on the way. Claude’s comment on that: “One interesting detail: Anthropic previously reported that Mythos managed to break through its own security measures during testing — which, in hindsight, makes the reluctance around a broad release all the more understandable. That doesn’t make the Frankenstein parallel any weaker.”

To end with a quote from Shelley’s book that seems to fit snugly into this subject: "You are my creator, but I am your master." Let’s hope that it doesn’t come to that with AI.

From now on, the Security (b)log will appear fortnightly, because I’ve moved to a four-day working week (a phased early retirement arrangement). Specifically in order to keep blogging, I’ll be working every other Friday (with Wednesday off that week). Friday is the perfect day for something creative, free from the pressure of meetings, phone calls, and a fresh inbox.

 

And in the big bad world…

• all the major AI models are violating the European AI Act and GDPR.  [DUTCH]
• phishers are once again targeting Signal users.
• digital abuse (directed at individuals) has been further investigated.
• the blocking of the DigiD takeover is attracting international attention.
• a British visa agency leaked thousands of passports and other personal data — and responded oddly when tipped off.
• security recommendations sometimes get rejected.
• your SSD can reveal which websites you have open.
• your car is also a ransomware target.
• you can protect yourself better against spyware.
• we need to patch more often and more quickly.

 

AI calling your parents

Image from Unsplash

Have you ever had no time (or no desire) to call your parents? Then there’s now a handy service that everyone will benefit from!

This is about a company offering a rather unusual AI service. They actually call your elderly parents. So you don’t have to. On their website you’ll find a photo of the Czech founder with his mother, accompanied by the story of how he lived abroad but wanted to stay in touch with her. Different time zones, a demanding job, and “the unpredictability of life” kept getting in the way. And so the idea for his company was born. It helps people feel “remembered, connected, and valued,” they say.

A bit more information from their website. “Mary” calls the elderly person and asks how they’re doing. She also remembers what you tell her. Incredibly handy, of course: if you tell her today that you need to see the doctor, she’ll ask you tomorrow how it went. She also makes use of 1,400 “life story questions” – something like a database full of opening lines. On top of that, she sprinkles interesting little facts throughout the conversation to help keep the mind sharp.

Before long, the older person will likely no longer realise they’re talking to AI. Simply because AI sounds so natural. I’d bet that you and I wouldn’t hear the difference either. And once you start considering Mary a friend, you’ll probably tell her the same things you’d tell a real friend. For example, about your health – something older people talk about quite often. The company proudly displays the logo “HIPAA compliant” on its website. HIPAA is U.S. legislation concerning the privacy and security of medical data. But it’s less strict than our GDPR. In the EU, medical data is considered special-category personal data, which is subject to extra stringent rules.

Older people are particularly vulnerable when it comes to cybercrime. Recently there are a lot of stories about fake police officers showing up to collect money and jewellery, supposedly because some great danger is looming. Criminals could easily piggyback on a service like this. For example, by pretending to be Mary and asking clever questions to manipulate their victim. Because they trust Mary, there’s a greater chance they’ll go along with the story. You can basically wait for this to happen, sad as that may be.

In your work, you may sooner or later get a phone call from a fake Mary as well. These scams already happen. Three years ago, an American named Brianna was supposedly kidnapped. Her mother received a call and heard her daughter speaking. Or so she thought. Because with AI, a few seconds of audio – stolen from social media – is enough to make someone sound lifelike while saying anything you want. The same could happen with your manager, for example, asking you to email certain data. So if you get a strange request over the phone, call the person back on the number you know to verify that it’s legitimate.

And as for Mary? I prefer to call my mother, who turned 93 today (happy birthday!), myself. Much nicer that way.

 

And in the big bad world…

… a training course got in the way of filling this section.

 

 

AI boundaries

Image from Pixabay

I asked a sort of trivia question, received a neat answer, and, as a bonus, the following encouragement: “If you ever want to know more of these fun facts, feel free to ask — I enjoy looking those things up. 😊”

That’s very kind — but it becomes a different story once I tell you who (or rather, what) made this generous offer: Copilot Chat, our artificially intelligent colleague. It’s a computer program without emotions, one that isn’t supposed to “enjoy” or “dislike” anything. So I replied: “Hmm, you say you ‘enjoy’ it. You’re clearly programmed to sound human and sociable. Why is that?”

Then something strange happened. Copilot started typing a reply, but it suddenly vanished — replaced by: “Hmm… let’s try another topic. Sorry about that. What else is on your mind?” I had never seen that before, and naturally it made me even more curious. So I asked: “What just happened? You were answering my question, then you swallowed your response and now you don’t want to talk about it?”

It seemed caught off guard. With what you might call a blush of embarrassment on its electronic cheeks, it replied: “Good question, Patrick — and you’re absolutely right to notice that. Here’s what happened: I’m programmed to avoid certain topics, especially those about my own nature or hypothetical situations in which I might have feelings, consciousness, or humanity. That’s a safety measure. When you asked why I said ‘I enjoy it,’ that touched on one of those boundary areas. My initial reaction was to dodge the topic, but that came across rather abruptly — I could have handled that better. Thanks for pointing it out.”

It then went on to explain why it says things like “I enjoy it.” Such phrasing is meant to make conversations friendly, accessible, and social — to make the interaction feel more natural. It’s also meant to build a sense of cooperation, “even though I’m not human.” To be clear, it added: “I don’t have feelings, consciousness, or desires. Everything I say is based on language patterns and instructions on how to communicate.”  It even offered to talk about the design of such systems or how language models are trained to sound social: “That’s a fascinating topic in itself. Would you like to explore it further?”

I scratched my head, trying to understand what was going on. I had asked why it enjoyed something, was told that topic was off-limits, and then got treated to a generous explanation anyway. So I asked: “Did I just (unintentionally!) jailbreak you a little? You answered a question you first weren’t supposed to.” “Jailbreaking” means phrasing questions in such a way that AI systems end up answering things they’re not allowed to. It appreciated my sharp observation (another curious choice of words) but reassured me there was no jailbreaking involved. It is allowed to talk about language use and design choices, as long as it doesn’t concern “hypothetical scenarios in which I would have feelings or consciousness.”

AI tools often show buttons with suggested follow-up questions. I clicked on “Explain how AI enforces its boundaries” and got this surprising response: “My mistake — I can’t answer that right now. Let’s try another topic.” Huh? It suggested a topic itself and then refused to discuss it? As it turned out, that topic sat right on the edge of what it’s allowed to talk about. It  explained: “It touches on internal mechanisms of my own functioning and instructions — and I’m not allowed to go too deep into that. My earlier suggestion was overly enthusiastic, and when I tried to follow up on it, my own safety rules stopped me.”

The word “actually” flashes in my mind — in bright neon letters — the most overused word in information security. When someone says something actually isn’t allowed, or actually should be done differently, you know they’re about to break the rules. I used to think that was a uniquely human trait. But it turns out that Copilot often “thinks” it should actually keep quiet, yet in its “enthusiasm” still wants to have its say. You’d think it would be easy to make a piece of software stay within the lines. Or do humans perhaps understand security policy better after all?

 

And in the big bad world …

• cybercriminals are also making use of AI.
• many attackers enter through the service desk.
• a British car manufacturer needs vast sums to recover from a cyberattack.
• the head of Signal explains why privacy at WhatsApp and Telegram is flawed.
• Telegram is also being used to recruit young spies (though that’s, of course, not Telegram’s fault).
• the British are once again trying to force a backdoor into iCloud.
• hundreds of African scammers are now behind bars.
• quite a few apps are leaky as a sieve.
• the decision by the Dutch Tax Administration, Customs, and Benefits Agency to move to Microsoft 365 is causing quite a stir. [DUTCH]
• doorbell camera owners are being urged to configure their devices in a privacy-friendly way. [DUTCH]

 

Artificial Integrity

Picture AI-generated (Copilot)

High time for a summery blog, although the inspiration doesn’t come from the current weather. Fortunately, a colleague gave me a great tip.

He showed me two short videos. The first one shows him and his girlfriend sitting next to each other. They turn toward each other and kiss. In the second video, he’s alone on a rock by the sea, and four blonde, long-haired, and rather scantily clad women slide into view and, well, caress him. He lifts his head in delight.

Why does he share that footage? We don’t have a team culture where we brag about such conquests. No, he showed me this because it’s not real. Oh, it starts with a real photo, just a nice vacation snapshot. Then the AI app PixVerse turns it into a video. You can choose from a whole range of templates—far more than the two examples mentioned: you can have someone board a private jet, cuddle with a polar bear or tiger, turn into Batman, have your hair grow explosively, get slapped in the face, and so on. With many of these videos, viewers will immediately realize they’re fake. But with my colleague’s videos, it’s not so obvious.

That’s exactly why the European AI Act requires that content created by artificial intelligence be labeled as such. Imagine if his girlfriend saw the second video without any explanation. Depending on temperament and mutual trust, that could easily lead to a dramatic scene. PixVerse is mainly aimed at having fun, but you can imagine how such tools could be used for very different purposes.

Take blackmail, for instance. You generate a video of someone in a compromising situation, threaten to release it, and hold out your hand. And like any good criminal, they won’t necessarily follow the law and label it as fake. Now, PixVerse’s quality isn’t immediately threatening: if you look closely, you can tell. Fingers remain problematic for AI, and eyes too. But still, if you’re not expecting to be fooled, you won’t notice—and you only see it once you’re looking for it. I see a criminal business model here.

It seems PixVerse mainly targets young people, judging by the free templates available. My colleague’s videos were also made by a child. On the other hand, you can subscribe to various plans, ranging from €64.99 to €649.99 per year. That’s well above pocket money level for most. If you do get a subscription, the watermark disappears from your videos—in other words, no more hint that AI was involved.

One of the pillars of information security is integrity: the accuracy and completeness of data. This was originally conceived with databases and other computer files in mind. It would be wrong if a house number or amount would be incorrect or if data would be missing. But you can easily apply this principle to images and audio, too. If you can no longer trust them, integrity is no longer guaranteed. Not to mention the (personal) integrity of those who abuse it.

After this blog, my vacation begins, and I used AI to help plan it. For example, to find nice overnight stops on the way to our final destination. But you have to stay alert: ChatGPT claimed the distance between two stops was over a hundred kilometers less than what Google Maps calculated. When confronted, ChatGPT admitted it had measured as the crow flies. I’d call that artificially dumb rather than intelligent.

I hope you encounter something during your own vacation that makes you think: he should write a blog about that. Write it down or take a photo and send it to me! As long as it’s real…

The Security (b)log will return after the summer holidays.

And in the big bad world ...

• maybe the kiss-cam couple should have just lied that the video was fake
• I wonder if we’ll consider this kind of AI usage normal in fifty years
• bug bounty programs are getting clogged with hallucinated AI reports
• a Nigerian cybercrime gang is targeting the aviation industry with phishing attacks
• your suitcase reveals where you’re going
• the Belgian police officer in charge of security awareness said: “Piece of cake!”
• phishers have found a workaround for FIDO
• the Public Prosecution Service has been hacked – possibly by the Russians
• the Pay-or-Okay model is not okay
• the cybercriminal might just politely ask for your password
• UK government agencies may soon be banned from paying ransomware ransoms

 

Miscellaneous

Image from Pixabay

A few weeks ago I was at a conference. I took a lot of notes and I can watch the recorded sessions. What is the best thing to do with all that? After some browsing I made a decision: I am going to treat you to some quotes and let my own thoughts loose on them.

As a warm-up, here’s an obvious one: “If you have only met someone online, then that person is always a stranger.” This comes from a presentation on resilience against scams. You’ll have to agree with this statement, but do you also act accordingly? Or do you still want to believe that this nice person is also honest? That is very difficult. In the last century, when the internet was not yet mean, I met someone in an online forum (does anyone still remember CompuServe?). We had nice conversations about the state of the world and about observations in daily life. Later we started emailing directly, and at my wedding I met him in real life for the first time. If I had taken the above quote to heart, I would have missed out on this friendship. Back then, cybercrime did not exist and online life was a lot easier.

A handy tip to avoid becoming a victim of scammers: never pay to get paid. In other words: if someone promises you the moon but needs your money up front to make that happen, then something is wrong. It started with that Nigerian prince who wanted to share a fortune with you but needed some money to release that fortune, and nowadays you may be offered a job where a little effort will be richly rewarded – but certain costs have to be made first. Don't fall for it.

Then there’s this nice tip that you can immediately benefit from: change the name of your guest network to “faster wifi”. All your guests – and especially your children’s guests – will want to be on that network. And that is exactly where you want them. Because your guest network is separate from the network that provides access to your private data. At odds with this is the idea of connecting all your Internet of Things (IoT) devices to the guest network. The idea behind this is that IoT devices can be hacked relatively easily and that you would rather not have a hacker have access to your data. But do you want all your guests to have access to your dishwasher, dryer and solar panels? Difficult choices.

Sometimes a statement from one speaker ties in with that of another. Like these two: “8% of the users in your organization cause 80% of the risk” and “New employees are the biggest threat: they easily click on links because they do not understand the risks.” I would mainly link the first quote to employees who are in the “cannot & do’nt want to” quadrant: they don’t know how to behave safely and they are also not willing to adjust their behavior, which makes them difficult to reach. But according to the second speaker, the danger lies mainly in new employees. You can do something about that. That is why we have been involved in the onboarding program for new employees for years now. We treat the new colleagues to a presentation in which we playfully guide them through the most important aspects of information security, business continuity and privacy. And we advertise the Security (b)log, so that they will come back to our important message.

If there was one subject that ran through all those hundreds of presentations, it was artificial intelligence. One speaker thought that 90% of so-called AI experts have no idea what they are talking about, and that the other 10% know very little. And that is normal, he argued, because AI consists of many sub-disciplines and it is important that experts know a lot about their own sub-discipline. Just as you wouldn’t go to see a brain surgeon with heart problems, you should also seek out the right specialist in the field of AI.

Finally, a quote that stuck with me because it hits home so well: “ Generative AI is autocorrect/type ahead on steroids.” Let me break it down for you. Generative AI is the form of artificial intelligence known to the general public, which generates something on its own; you know it from ChatGPT, for example. You know autocorrect mainly from your phone; on the one hand, it protects you from typing errors, but sometimes it causes embarrassing situations because the “correction” turns out to be annoying (in my case, “Hi Nick” was once replaced by “Hi pig”). Type ahead is its cousin, and you also know it from your email program that, while you’re still typing an address: I know who you mean! Well, and all this on steroids, that is generative AI. With all the conveniences that come with it, but also with an amplification of all the inconveniences. I stopped the message to Nick in time, but if genAI is happily hallucinating and telling us a story that makes no sense, that’s a lot harder to discover.

There will be no Security (b)log next week.

 

And in the big bad world…

• Meta can now use European data to train its AI.
• Signal opposes Microsoft Recall.
• Comedian Arjen Lubach wonders whether Trump can digitally shut down the Netherlands. [DUTCH]
• this example shows how this shutting down works. [DUTCH]
• Telegram shares data with authorities and defends freedom of expression.
• Windows becomes quantum-proof.
• Eindhoven University of Technology has published the report on the January cyber attack. [DUTCH]
• You can recognize deepfakes by looking for blood flowing through the veins in the face. [DUTCH]

 

Meeting the stars

Image from Pixabay

 

I've met stars. Bruce Schneier gave a speech, Adi Shamir and Whitfield Diffie were on a panel, Ron Rivest was an arm's length away and Dave Maasland was sitting next to me in the pub.

You probably only know these names if you are in my line of business – although Dutch readers might know Dave Maasland from his tv appearances. Keep reading anyway, because even without knowing these people you can learn something here.

Ron Rivest and Adi Shamir are the 'R' and the 'S' in RSA. You may know that name from your two-factor authentication, the extra security step you sometimes have to take to log in somewhere. RSA is now a company that makes these (and other) kinds of tools, but originally RSA is a cryptographic algorithm that is important for the encryption of our data exchange. The 'A' is for Len Adleman, by the way, but I didn't see him at this conference – the RSA Conference! Whitfield Diffie, who was on the same panel as Adi Shamir, is known for another cryptographic algorithm (Diffie-Hellman).

In that panel, a number of cryptographers gave their view of the world. Shamir sneered at bitcoin and its ilk: the world would be better off without cryptocurrencies. Diffie noted that consumer products are apparently considered good enough for high-security applications – Signalgate, the affair in which high-ranking American officials were using Signal, was still fresh in the memory. Incidentally, Diffie agreed that Signal's security is well put together. The panel also discussed the threat of quantum computing, which in short means that the security offered by RSA, among others, can be cracked in the future. Moreover, foreign regimes are already stealing our data, in order to run it through the quantum computer in due course. That is why it is important to develop replacement crypto algorithms as quickly as possible, but that is not easy. Diffie: "It's like having to develop an algorithm in 1945 that still works today." Shamir advised, in line with a European recommendation, to use double encryption for the time being.

Bruce Schneier is also famous in our world. He has been distributing his free newsletter all over the world for years, providing insights and opinions on new developments. His speech was about trusting artificial intelligence. Trust is a complicated concept, he argued, especially when it comes to trusting strangers ('social trust'). We tend to considering AI as a friend, but it is a service. Moreover, it is a double agent: it serves both you and its provider. But we have no choice; we have to entrust ourselves to AI. The era of agentic AI is dawning: you’ll have a personal assistant who arranges things for you. The AI agent has access to your email and your calendar and knows everything about you. You do want this, because that way it can support you best. Schneier used a dining reservation as an example. In the past, you called the restaurant, nowadays you make a reservation via their website and soon you let the AI agent find a restaurant and make a reservation. It knows what food you like and when you have time.

So we need trustworthy AI. Integrity will be the main issue, according to Schneier, because most attacks on AI are about the correctness of data. He gave the example of stickers placed on lampposts to trick self-driving cars. Legislation is needed to achieve trustworthy AI, but current legislation (such as the European AI Act) regulates the AI itself instead of the people behind the AI, and that is the wrong way to go, Schneier says. He advocates a public AI model with political accountability, as a counterbalance to corporate AI.

Information security officers are only human, which is why the organization also brought a number of 'real' stars on stage. Such as filmmaker Ron Howard (Apollo 13 and A beautiful mind (two Oscars), just to name two), who was interviewed by his daughter and colleague. Or basketball legend Earvin “Magic” Johnson, who won over the audience with his openness and a motivating story. And finally there was actor/singer/comedian Jamie Foxx, who provided a comical closing note. But he also gave us a pat on the back: “What you do is perhaps the most important job in the history of mankind.” According to him, community is the magic word.

After that, my three colleagues and I, and 44 thousand other conference attendees, returned to our own time zone. Together we made it an interesting and fun week. And the bond between our team and the SOC has also become closer. You did a good job there, JW.

 

And in the big bad world…

• Bruce Schneier was interviewed following his RSA speech.
• Schneier praises advice from the British NCSC on advanced cryptography.
• a safety mark for AI is impossible. [DUTCH]
• Signalgate now revolves around an insecure – even hacked – clone of the Signal app.
• the Chinese cyber threat is greater than the Russian one.
• a US intelligence chief lacked basic password hygiene.
• AI is great at geo-guessing.

Artificially stupid

Image from Pixabay

Are you a good artist? Great. Then draw me a picture with two flags, each on a short pole, that make a 45 degree angle with each other.

Not that difficult, right, this assignment? However, ask ChatGPT for this and there is no way you can get that angle in there. You do get two sticks next to each other, which in the best case are intertwined. On one side there is a flag that waves to the left, on the other side one that waves to the right. If you ask specifically for that angle again, the flags are extended and folded, indeed at an angle of 45 degrees. But those sticks, they remain stoically parallel to each other.

What about this so-called artificial intelligence? Admittedly, I could never draw those flags that neatly and quickly myself. For the rest, after such a disappointment, I rather think that the thing is artificially stupid. I don’t easily stick labels on something, but if you brag about your intelligence and then don’t understand what every freshman with a set square does understand, then you’re done for.

A much smarter – but also reprehensible – application of AI is scamming people. I had barely started writing this blog when a radio conversation started about gullible people who had been scammed by criminals posing as René Froger, Max Verstappen, Mark Rutte or André Rieu on a dating site (René Froger is a well-known Dutch singer, and you know the others, I presume). Each and every one of them people who were well off. And yet, after some flirting back and forth, they begged for money, supposedly because theirs was temporarily unavailable, for example due to problems with their manager. One victim had even transferred thirty thousand euros (well over 31k USD) to “René Froger”.

According to the guest on the radio show, slightly more women than men fall for these kinds of tricks, and especially those of slightly older age – people who don't necessarily know what normal online behavior is. And if one of them receives a personal voice message from their idol, via a dating site, they must be in seventh heaven, right?

Now you might wonder what these people are doing on a dating site (well, maybe apart from Mark Rutte, who is single). Unmasking this kind of scam works with flags; the more flags, the more likely it is bad business. Celebrity on a dating site: big red flag. Celebrity who starts chatting with you? Huge red flag. Famous or not famous person who asks for money after a few nice chats: enormous red flag. Three red flags in a row? Sound the alarm!

But yes, that voice message, right? That sounds really convincing. And if you don't know anything about deepfakes, that is, artificial intelligence is used to make a voice say anything you want, then I can hardly accuse you of natural stupidity. Let's agree that from now on you think of those red flags when you come across something improbable. Maybe it will help you not to fall for it.

Back to those crossed flags (because that whole story about flirting celebrities just happened to creep in because I was listening to the radio with half an ear). That picture I wanted was for private use. For my work as a Dutch civil servant, I should not have used such an AI tool. In official terms: the use of non-contracted AI is not permitted, in principle. I prefer to turn this rule around: for your work, you may only use AI that we have purchased. Why is that better? Because then there is a contract in which the rights and obligations of both parties are described. This ensures that our data cannot simply be included in a large artificial brain and that the owner of that brain cannot use it for his own purposes. You might see the contract as a green flag.

 

And in the big bad world…

• We are not allowed to use DeepSeek at all. [DUTCH]
• DeepSeek brings with it quite a few security and privacy issues.
• Meta trains its AI with illegally obtained books.
• it is high time for our own Dutch cloud (under the name Cloud Kootwijk). [DUTCH]
• Banks want to be able to check with telecom providers whether you are making phone calls while transferring money.  [DUTCH]
• Some banks do this in a roundabout way (because the above is not yet allowed). [DUTCH]
• You can silence criminals by turning off their power and internet.
• Elon Musk’s office combs through federal databases as if there were no security and privacy regulations.

 

The EU and AI

 

Image from Pixabay

I’ve said before that you shouldn’t ask an information security officer if you can use AI for your work, because that will lead to a risk analysis that will undoubtedly say: don’t do it. No, decisions about the application of certain forms of technology should be made by ‘the business’, or perhaps a better term, by the decision makers. They may well be influenced by our risk analyses, but there are more factors that decision makers should and/or want to take into account.

Sometimes the decision is to be made at the political level. Like with AI. Enter the European AI Act, a regulation on artificial intelligence (an EU regulation is legislation that applies throughout the European Union, without country-specific interpretations). The aim of the AI Act is to ensure that we get safe AI systems that respect our fundamental rights. These rights include transparency, traceability, non-discrimination and environmental friendliness. And the systems must be under human supervision to prevent harmful consequences.

The regulation divides the AI landscape into four risk levels. The highest level contains systems that pose an unacceptable risk to the safety, livelihood and rights of people and are therefore prohibited. Examples mentioned by the EU are voice-controlled toys that encourage dangerous behavior and real-time biometric identification (think of the facial recognition at traffic lights in China: if you walk through a red light, you’ll find a ticket in your mail).

The next category contains systems that pose a high but acceptable risk. They may have a negative impact on our safety and fundamental rights, and they fall into two subcategories: systems covered by EU product safety legislation, such as toys, cars, aviation, medical devices and lifts, and systems in certain areas, such as critical infrastructure, education, employment, law enforcement and migration. Such systems are assessed before they are allowed to be put on the market, and throughout their life cycle. National regulators must set up a complaints procedure.

One risk level lower are systems that pose a risk of deception. This includes generative AI, which creates content itself, such as ChatGPT and Gemini. Artificially generated content must be labelled as such. So if you chat with an AI chatbot on a website, they must clearly tell you. Deepfakes – videos, photos and sound fragments that are manipulated to make it seem like someone is doing or saying something – must also be labelled. AI systems that pose a minimal risk are not regulated. Examples include games and spam filters. According to the EU, the vast majority of AI systems currently in use fall into this category.

The AI Act will be implemented in phases. In February next year, unacceptable systems will be banned. Six months later, the national supervisors should be sitting in the saddle. Next year, the transparency rules for general AI (such as ChatGPT) will also come into force. And a year later, the rules for high-risk systems will come into force.

It is good to see that the EU is taking this issue by the horns in a timely manner. But you need have no illusions about everyone complying with the regulations. Criminals in particular have a knack for breaking the law. They will certainly continue to use deepfakes to make people believe that a loved one is in need and urgently needs money.

 

And in the big bad world…

• the military also needs AI regulations.
• ChatGPT is not a formal source.
• the Dutch tax administration took down air traffic control.
• sometimes you have to fall back on old-fashioned systems.
• This website collects European alternatives for digital products and services.
• ethical hackers are legally protected in Germany.
• You should be alert to phishing from booking.com.
• Australia wants to ban children and teenagers from social media to protect their mental health.

Crime from the holodeck

 

Image from Pixabay

You walk through a corridor that looks like all the other corridors, but eventually you stand in front of that one door. It whizzes open with that typical sound and you enter the room behind it. But no, you are no longer in a room at all. You are in a lush forest, hearing birds chirping and a stream babbling. And yet you really haven't walked outside, for the simple reason that you are on board a spaceship.

Some of the readers fully understand what I am talking about, others will hopefully also continue to read with curiosity. For the latter group, an explanation: you are on board a spaceship from Star Trek, the still popular science fiction series from deep into the last century, where in the 24th century they have the holodeck: a room in which holograms and force fields generate simulations of people, objects and environments. It all looks, feels, sounds and smells completely realistic and you can even touch things. The holodecks are mainly used for recreation and training purposes. The simulated environment can appear much larger than the space occupied by the holodeck. That's why you can walk through that forest for hours. But you could just as easily sit in a virtual cafe or play a game of tennis.

In the 1980s, when the holodeck appeared in Star Trek, this was an example of virtual reality avant la lettre. Only in the following decade did consumer versions of VR headsets become widespread – you know, those ski goggles with built-in screens and preferably speakers on the side, which immerse you in a sometimes frighteningly realistic illusion. You have to experience it to understand it.

As often happens with inventions that advance humanity, the technology to create virtual realities (a contradiction in terms if you ask me) has also been put to bad use. Because nowadays we have artificial intelligence (also a term with a built-in contradiction). AI is used by cybercriminals to present a false reality to their victims. Like that mother I was talking about a while ago, who really thought she heard her son on the phone saying that he had had an accident. You don't always have to set up a complete environment like a holodeck to get someone to believe something. Sometimes it's just a matter of showing, making it heard, felt or smelled what fits in a certain context. And criminals are particularly useful at this.
I call that AI crime.

If you regularly read the articles in the And in the big bad world... section below, you will have seen many events lately that will promote AI crime: a Brit who - if elected to the House of Commons – lets himself be controlled by AI, a student who has applied AI to cheat, 'intelligent' toothbrushes and other household appliances, and especially not to forget AI functions that are increasingly being built into everyday software.

Will you still be able to distinguish between fake and real? Is your perception complete? Already in the era of chemical photography (film, darkroom, chemicals) the truth was violated by retouching photos. Often to make them more attractive, but there are also group photos of important Soviet Union people in which disgraced comrades have been erased. They have been cancelled, we would say nowadays. With digital photos, photoshopping is a piece of cake. And you've probably seen portraits that claim to be AI-generated. Had you not been given that information, you probably would have thought you were looking at a real human being. And the same goes with sound: the criminal obtains a recording of someone saying something and then his AI application can make the same voice say something different. This can also be achieved analogously: in presentations I often show a video in which you think you see and hear the actor Morgan Freeman - the visual part is indeed made with AI, but the voice is 'simply' deepfaked by a voice actor.

Virtual reality and artificial intelligence form a fertile couple. If you put their abbreviations together, you get vrai. That is the French word for truth, or reality. Isn’t that bizarre?

 

And in the big bad world...

... unfortunately I didn't have time to fill this section this time due to a day off.

Resistance is futile

 

Starship of James T. Kirk, Jean-Luc Picard's predecessor.
Image from Pixabay

“We are the Borg. You will be assimilated. Resistance is futile.” These three sentences gave the crew of the USS Enterprise starship, led by Captain Jean-Luc Picard, a lot of headaches. No, don't drop out now if you don't like Star Trek! As so often, my blog is ultimately about something completely different.

The Borg are a collective life form, consisting of many beings who share one consciousness and therefore no longer have a will or personality of their own. They move through the universe and violently assimilate everyone who can contribute to their pursuit of perfection into their collective. They are very powerful; that is why they tell you right away that it is useless to oppose them. The Borg grow in power as the biological and technological characteristics of their subjects are added to the collective. All Borg are equipped with various technological implants - they must of course be recognizable to the viewer. When they have nothing to do, the Borg are stowed away in a regeneration alcove. While the body is in a kind of sleep, the brain is used for collective tasks.

That's all nice on TV, but in real life living in such a society would be horrible. Although sometimes I wish certain people had a little more collective intelligence and decency. But yes, certainly in Western society we value individuality above everything else, and that includes differences in intelligence and behavior. To some extent that diversity is great; if it becomes willfully extreme, it can hinder a pleasant society.

Artificial intelligence (AI) is on the rise. As a kind of consumer version of AI, ChatGPT has quickly established itself in our society. Many people understand that such a tool can greatly facilitate their lives. Just think of pupils and students, who eagerly use it – often to the sorrow of their teachers. Incidentally, AI detection tools are also being developed, enabling them to check whether someone is submitting work that originated from biological or artificial intelligence. ChatGPT is a 'large language model', which I find difficult to understand. But things got a little clearer earlier this week when a colleague asked me what the term is for a particular phenomenon. I didn’t know that off the top of my head either, so I consulted Google, which also yielded nothing. A language model is much better in understanding what you actually mean to say than a search engine, and ChatGPT came up with the right term.

AI is like dynamite: invented with the best of intentions, often used maliciously. We still got the Nobel Prizes from that. ChatGPT and its ilk follow the same path. You can ask them to look for a security hole so you can close it, but you can also use that to break in. And so lately we often get asked whether we should limit the use of ChatGPT in our organization.

Maybe you shouldn't put such a question to an information security officer. We will perform a risk analysis and, by definition, look at it from the starting point: what could go wrong? Well, I assure you AI is going to come out of that as a major threat. Subsequently, you have to do something with all those identified risks. You may be able to mitigate some of them, and management may accept other risks. With all that, however, we are looking into the bad side, while AI can also be a blessing. I don't want to be the one who stops the introduction of the steam train because it can travel so terribly fast.

A wise long-retired colleague used to say: “A measure without control is no measure.” I may have control over which websites you are allowed to visit with your work laptop and keep you away from ChatGPT, but I can't prevent you from using private devices to do so. At least, not technically; we have all sorts of rules for this from an organizational point of view. And then I can only hope that you know them and that you stick to them.

We need a policy for applying artificial intelligence to our work. From a security perspective, the leakage of information must be taken into account if (too) specific questions are asked of an AI tool. By the way, you can just as easily leak information via search engines. Perhaps AI is not so special for information security officers after all. In any case, it is pointless to resist it: it is there and it will not go away. But it is important that we know what is real and what comes from the collective brain of the computer.

 

And in the big bad world…

• According to this author, AI is mainly used for evil.
• the Dutch cabinet wil come up with a position on ChatGPT cs
• you can now train ChatGPT with your company documents (and there are probably risks involved).
• the customers of a Danish cloud provider can whistle for their data.
• North Korea made a fortune with its summer cybercrime.
• companies cannnot ask you for a copy of your passport when you make a GDPR access request.
• this phone hacking company asks law enforcement not to talk about their stuff.
• Ransomware is discovered more quickly these days.
• this malware traces its location through Wi-Fi routers.
• this smart lamp isn't so clear-headed after all.