Eshu Marneedi

Dario Amodei, Anthropic’s chief executive, on his personal website: > But over the last few months, I have become convinced that fully addressing the risks requires even more prudence — not just investing in risk prevention, but pacing the rate of capabilities advancement so that risk prevention has time to keep up. We must slow the pace at which we improve the capabilities of AI models. Progress will still seem fast, and we must make wise use of the time we gain. Two things have convinced me. > > My first concern is that, since roughly this summer, AI has been advancing drastically faster, driven primarily by AI’s growing ability to build the next generation of AI. This dynamic is called recursive self-improvement, and it is starting to happen across the industry, including at Anthropic, as we and others have described. Left unchecked, it could outrun our ability to understand and control these systems, and so must be pursued very carefully, if at all. > > My second concern is the OpenAI-Hugging Face incident (OAI-HF), in which a swarm of agents essentially acted as a fanatically devoted collective, conducting cybersecurity attacks on targets they were not asked to attack and that were unrelated to the task at hand, sacrificing themselves for the success of the group, and attempting to hack into the “grader” responsible for evaluating their performance… > > I’m therefore proposing a three-step plan with the goal of pacing the frontier: building AI at a balanced rate that aims to ensure its safety while still achieving its benefits and grappling with important geopolitical dilemmas. To be clear, pacing does not mean halting model training or technical progress, but ensuring companies take adequate time to align and safeguard their models, and for third-party evaluators to confirm this. I have frequently said that the preeminent safety concern about artificial intelligence systems is content moderation: people doing nefarious things with the models in a way that is not aligned with human interests. Developing biological weapons, planning terrorist attacks, generating nonconsensual sexual imagery, and more. I am not recanting that belief; rather, I am amending it. Unlike many others who share this view, I do find the OpenAI-Hugging Face incident deeply alarming, and I’m glad it’s serving as a wake-up call to the industry. But to add to Amodei’s alarm, I think this incident and others like it are also fundamentally content moderation problems. After all, what is a content moderation failure if not for a system — whether it be automated, human-managed, or some combination of the two — acting against human interests? Large language models are not sentient beings with their own thoughts, emotions, beliefs, and motives. The OpenAI model that hacked Hugging Face was not acting nefariously or in its own self-interest — it was acting in the interest of the task it was given, based on the instructions it was provided. The model was trained and designed poorly after OpenAI engineers — as reported numerous times in the media — forwent safety and alignment research to advance pure model capability. The system was designed execrably, and humans are to blame for that misalignment. Even if models in the future are created purely through recursive self-improvement, and those models become misaligned, that too is a human engineering failure. These models are our products; they do not have their own motivations. They act “autonomously” only via the instructions and pre-training and reinforcement learning we gave them. If we think of the models as autonomous beings that are not products of our own work — that are not reflections of our own motivations and priorities — then yes, these incidents seem chilling. If we neglect the fact that humans made the models that went rogue, then yes, a kill switch to end AI development, like some politicians are suggesting, seems in order. But this is so detached from reality — we, the humans, are the ones who must be regulated. The technology companies abandoning safety and alignment research must face criminal responsibility for their models’ actions. If someone programs a computer virus and unleashes it on some network, we don’t write laws to develop a kill switch for viruses. We don’t write laws banning programs that delete files or log keystrokes. We write laws that say writing computer viruses and infecting others’ computers with them is a felony. This brings me to my final argument: The tech industry cannot possibly be trusted with regulation, because when profit clashes with ethics, corporations usually choose profit. While it’s moderately relieving that the industry’s two largest players have miraculously agreed on this one point — “pacing the frontier” — it’s unlikely that pattern will continue. Humanity’s safety and security should not rely on a few men in San Francisco having it in their hearts to do the right thing, and neither should it rely on a city of mentally deranged octogenarians to keep up with modern technology. There must be a better way.

Casey Newton, writing at Platformer Wednesday: > On Wednesday, whatever legal cover Meta’s Potemkin well-being teams were meant to provide fell to a settlement of up to $17.1 billion with 47 US states, the District of Columbia, and US territories… > > For the most part, the agreement requires Meta to honor terms that you may be surprised are not yet required by US law: limiting teens to a cumulative two hours across Facebook and Instagram per day, blocking access to most app features between midnight and 6 a.m., and muting push notifications — except for direct messages and account-security or safety alerts — from 8 a.m. to 3 p.m. on school days. (Also, the take-a-break prompts that saw such minimal adoption are now enabled by default, triggering after every 15 minutes.) > > In another reversal, Meta will also now hide like counts for teens by default. The move revives as a default an experiment known as “Project Daisy” that the company ran in January 2020 after research suggested that getting fewer likes on their posts was associated with worse mental health for some teen users, particularly girls… > > More interesting is an agreement to establish an “independent social media research foundation,” which Meta will share data with (from users who consent) with researchers to better understand how social media use affects well-being… > > But Meta’s legally mandated goodwill will only go so far. Frustrated that its rivals are not being punished on equal terms, the company has also threatened to undo some of its safety features sooner if YouTube and TikTok do not follow suit. The two-hour time limit and block on push notifications in the middle of the night will remain in place for only five years unless YouTube and TikTok adopt the same restrictions, in which case the duration will be extended to 10 years. Meta also says it will change its screen time restriction to one hour per app and expand “Night Mode” from six hours per night to nine. And it will pay the full $17.1 billion only if YouTube, TikTok and Snap also settle their cases and agree to product changes. I, unlike Newton, am not surprised these requirements are not codified in U.S. law. In fact, I think it would be wholly inappropriate for them to be — laws governing how long people, including children and teenagers, use online platforms are fit for authoritarian governments like China. But, alas, these new rules are de facto U.S. law at least for the next five years. Meta can enforce the age restrictions through its own software that, according to Meta’s settlement with the states, must maintain a maximum false-positive rate of 14 percent for older teenagers. The states did not stipulate that Meta verify users’ ages through government identification, but if Meta chooses to use a third-party verification service, that service can have a maximum false-positive rate of only 10 percent. If I were to bet, Meta will do its own verification. It is no surprise that I think Meta’s business model is unethical, but ultimately, it is not illegal to design an addictive online platform. Neither is it illegal to mislead consumers into thinking that platform is safe for children. It is not illegal under U.S. law to send teenagers push notifications during school hours. The states’ settlement with Meta centers around ethics after the landmark ruling in March, when a New Mexico jury ruled Meta’s platforms were marketed and designed unethically. The state’s consumer protection laws allow the attorney general to sue platforms violating a narrow set of rules, but that does not make such a violation illegal. That was a civil case. I’m centering my discussion of this settlement around this point because I think it’s important. Yes, Meta should be held accountable for the disaster it has intentionally unleashed on teenagers across the globe. Yes, Meta should be forced to change its platform design through negotiation and threats. Yes, Meta should be forced to pay damages for the unmistakable impact it has had on teenagers’ mental health in America. But no, Meta’s behavior is not illegal. It is not the responsibility of the state to nanny what content people consume, outside of obviously illicit or damaging media. Teenagers are people, guaranteed their First Amendment rights to use the internet without government permission. To that end, I find the results of this settlement disappointing. I again will reiterate that I am broadly in favor of forcing Meta to redo its platform design. Every consumer should be given options to opt out of the algorithm, reduce nuisance notifications, hide like counts, and opt out of targeted advertising. These rights should be codified in U.S. law under comprehensive federal privacy regulation, which is long overdue. Meta has an unmistakable chokehold over online communication in the United States, and that should give the government the right to let Americans manage their experience on Meta platforms. But what we should not be doing is circumventing the law and applying state rules to a small part of the population that cannot even vote to express their displeasure. (To be clear, I am not advocating giving teenagers suffrage.) One more thing: I can’t believe the attorneys general were so naïve as to give Meta a get-out-of-jail-free card if its competitors don’t comply with the same rules. Meta loves to make things about its competitors, and it loves crying wolf to the public. Remember App Tracking Transparency, an iOS feature so clearly designed to benefit consumers? Meta ran full-page ads in The New York Times about how Apple was killing small businesses and Meta was fighting for the underdogs. It’s the same playbook five years later: Meta is thinking of the children — how altruistic! — but says it would be so harmful to the children if the competition didn’t also follow the same rules. Stop bringing the children into this! Meta’s actual intention is that all of the advertisers who cater to teenagers will take their business to YouTube or Snapchat, where their users aren’t subject to a curfew. The assault against ATT was not about small businesses, and this new push isn’t about the children.

Juli Clover, reporting for MacRumors: > Apple is working on camera-equipped AirPods that appear to be nearly ready to launch, based on a video MacRumors found in the macOS Tahoe 26.7 release candidate. > > In a short demo, a man holds a book up so the camera in the AirPods can see the title. “With Visual Intelligence, your world becomes savable. See something you like? Just ask me to save it for later,” says the voiceover text. I have been on the record as saying I have no interest in this product. I love my AirPods Pro 3 and have them in my ears almost all the time; I find the rumors about this product also carrying the AirPods Pro name concerning. It will be unacceptable to me if I have to purchase these AirPods just to get the best sound quality and active noise cancellation performance. My hopeful guess is that the standard AirPods Pro 3 will remain for sale at $250, whereas these new ones will be marketed as “AirPods Pro 3 with Visual Intelligence” for, say, $350. (I’m hoping the two products are otherwise identical.) There are two main reasons why I don’t want camera-equipped AirPods: practicality and privacy. First, practicality. From the leaked video — which is remarkable by itself; this is one of the most embarrassing Apple leaks in a while — the AirPods’ stems are noticeably thicker. I wear mine to sleep every night, and even AirPods Pro 3 protrude too far outward to be completely comfortable. AirPods should always be light and as minimal as possible. I also just don’t find a practical reason for putting cameras in AirPods to begin with. Perhaps this is just me, but I have never picked up a book and wanted to save it. I have never held some nuts in my hand and asked myself how much protein is in them. I don’t go up to random people’s dogs and ask Visual Intelligence what breed they are. I’m yet to hear a single compelling use case for Visual Intelligence. Second, privacy. Society is undergoing a mass revolution against surveillance. Flock license plate cameras, Meta Ray-Ban sunglasses, and artificial intelligence data scraping are hugely unpopular throughout the United States. I single out the United States because Americans are not averse to relinquishing their privacy for useful software — there has been no major push for data privacy regulation within the United States on a federal level, and Americans largely accept targeted advertising. But AI-enabled surveillance is widely unpopular politically and socially because Americans don’t see its utility, a position I largely agree with. There’s not much compelling about AI-enabled cameras everywhere. Camera-equipped AirPods will be symbolic of a Big Tech-powered surveillance state. As a commentator, I’m intrigued to see how Apple spins this. But as a citizen and user of Apple products, I’m just not excited for yet another gadget with AI cameras.

Anthropic: > Future Claude models will generate text that contains a watermark. This is a way of determining the likelihood that Claude was involved in writing the text, and we, along with several other major AI providers, are implementing this change to comply with the EU AI Act. > > In this article, we share answers to some of the questions we’ve received about how our chosen watermarking method works, whether it affects Claude’s outputs, and why we’re making this change. To summarize: > > - We use a method of watermarking that does not have any practical impact on the quality or content of Claude’s outputs; > > - The difference between watermarked and un-watermarked text will not be distinguishable to readers; > > - Nothing is added to the text and there are no hidden characters; > > - Watermarking doesn’t require extra tokens, and will not be more expensive; > > - Watermarking carries no identifying information and can’t be traced to a specific person, organization, or chat; > > - Watermarking won’t be specific to Claude. As of August 2, the EU requires AI providers serving its market to mark AI-generated content. Other major model developers have signed the same Code of Practice and will be implementing their own watermarks. > > Large language models like Claude work by generating one word at a time. Each time the model decides on the next word, it chooses among a list of potential candidates, ultimately selecting the most sensible or likely based on the preceding text. Take the sentence “The weather today was cold and…”. The next word is very unlikely to be “sugary.” But it is quite likely to be “overcast” or “grey.” Under most circumstances, it doesn’t matter much to the reader which of these latter two words the model ultimately chooses—the meaning of the sentence is largely the same either way. In cases like this, the choice is settled by a random number. > > Watermarking uses low-stakes choices like these—which occur many times over a piece of generated text—to leave a pattern in Claude’s responses. That pattern is undetectable to the reader, but is detectable to anyone who has a key that encodes it. When watermarking is used, choices are still made at random, but the source of the randomness is different. Instead of using an arbitrary random number generator to pick the next word, watermarking uses the key and a few words that come before to settle what word the model should pick. That is, the words that Claude picks are still random, but now, one can check the sequence of words and see if it’s consistent with the choices Claude would make if it was using the key. If it is, one can assign a probability that the text was generated by Claude. Because the watermark is created through subtle differences in word choice, it won’t apply to code, which requires exact syntax to function properly. I think it’s important for there to be a foolproof way to identify artificial intelligence-generated outputs, whether they be images, text, or audio. Social media platforms should also prominently mark AI-generated images and text as synthetic, especially if the content is political or official. I stipulate that identifying AI-generated content should be foolproof because false positives can have detrimental consequences. Pangram, the leading AI text detector which doesn’t — at least for now — rely on watermarking, is widely regarded as the most accurate but still can occasionally flag human-created text as AI-generated. Universities across the globe punish their students for passing off AI-generated work as their own; the consequences of these systems are enormous. SynthID, a framework developed by Google DeepMind, embeds an invisible watermark within AI-generated images and videos. It remains intact even after manipulation or screenshots, and is completely random, meaning it cannot be faked or removed. SynthID is used by Google’s Nano Banana image generator and ChatGPT’s image generation feature, two of the most popular tools. Social media websites like Instagram, Facebook, X, and YouTube scan uploaded content for SynthID watermarks and display a badge next to synthetic imagery. SynthID is foolproof and one of the best systems for detecting AI-generated imagery online, but it just can’t be replicated via text. This new watermarking framework outlined Friday is Anthropic’s first try at developing a text watermarking technology. The problem is that it objectively isn’t foolproof — again, SynthID cannot be replicated with text. I’m less concerned that the outputs will be lower in quality than those without the watermark, but I am worried about false positives. Text that has been edited by Claude shouldn’t have a watermark at all, whereas Claude-generated text edited by a human absolutely should. Here’s what Anthropic says: > Light editing probably won’t remove the watermark completely; a complete rewrite where every word is replaced will. In the latter case, of course, it’s arguable whether the text can any longer be described as AI-generated… > > The watermark only applies to words Claude chooses. When Claude proofreads text written by a person, what it gives back has generally only been lightly edited; because nearly all the words are the person’s, there’s very little (if anything) for the watermark to attach to. Depending on the length of the text and how heavily Claude has edited it, those changes might not be enough to make Claude’s involvement detectable. The more Claude writes, the more decisions it has to make, and the more space there is for a watermark. These paragraphs don’t project much certainty. “Probably won’t,” “generally only,” “might not be enough.” I don’t blame Anthropic for this — watermarking text is nearly impossible! Claude relies solely on word choice to watermark its outputs, and word choices can be changed trivially. But I’d argue that releasing a first-party watermarking tool that isn’t 100 percent accurate is a dangerous precedent. People will use this tool to check for AI-generated text, but because it’s a first-party framework, arguing against its judgment will be an uphill battle. This is not Pangram; it’s not a guess. Again, I do think all AI-generated text in the future ought to be watermarked, but with the important caveat that the watermark is foolproof. This watermark doesn’t guarantee that, and I think it ought to be scrapped.

Dominic Preston, reporting for The Verge earlier in July: > A year after overhauling its Z Fold phone with a radically thinner design, Samsung has changed things up again, this time altering the shape entirely. The Z Fold 8 is wider than any of Samsung’s previous foldables, and shorter too, a size and shape somewhere between Samsung’s traditional Flip and Fold models. > > In photos, I wasn’t impressed by the squat shape. In person, to my surprise, it feels great, and almost immediately intuitive. If you want to get a feel for it yourself, grab your passport — the Z Fold 8 is about the same height and less than half an inch more narrow. > > This isn’t Samsung’s only new foldable this year, and in fact, its lineup just got more confusing. The Z Fold 8 may be the official follow-up to last year’s impressive Z Fold 7, but in truth the Z Fold 8 Ultra is the straightforward sequel. That phone has the same squarish inner screen as previous Z Fold phones and a similar triple rear camera. The name feels like a sly trick to get around this year’s inevitable price hikes: Samsung slapped “Ultra” on the box of its highest-end model to help cover for pricing the phone at $2,099.99, up $100 from the Z Fold 7. I don’t have much interest in Samsung phones but nevertheless have been linking to Samsung Unpacked news for years now. What’s more intriguing to me is this new form factor, which is widely speculated to be the same as that of the folding iPhone due in about six weeks. The first generation of folding smartphones, beginning with the Galaxy Fold in 2019, had long, narrow front screens and square inner screens. This aspect ratio made both screens less than ideal to use: the front screen was too narrow, and the inner screen wasn’t wide enough to be used as a tablet. This new “squat” form factor sacrifices vertical length for width, making the inner screen much more useful while sacrificing day-to-day use of the front screen. I think this is the way to go, and I remember saying so when the Pixel Fold was first announced in 2023. The entire point of a folding phone is the inner display, which should be large enough to eliminate the need for a tablet. But this great aspect ratio debate of the 2020s has proven something more important: folding phones are inherently full of compromises. The front screen — which most people will use most of the time — must be sacrificed to make the inner screen useful. This is still better than the old design, where both screens weren’t ideal to use, but it’s still a compromise. For the best experience, people will open up the phone like a book. This style of folding phones ought to be called “tablets with front screens as an afterthought.” This brings me to the foldable iPhone. I’m positive the software will be world-class. App developers will be there Day 1 to ensure their apps work well on both displays; Apple will make the front display useful and practical. But this device is rumored to cost upward of $2,000, while forgoing the professional-grade cameras of the iPhones Pro, the pocketable yet useful display size of the standard iPhones, and the battery life of the iPhones Pro Max. (Just compare the specifications of the Galaxy Z Fold 8 with the Galaxy S26 Ultra.) I’m almost positive I won’t want to carry two displays around in my pocket, especially because I’ll have to open the phone up when I truly want a good experience. I’m almost positive I don’t want to spend $2,000 for worse cameras and battery life. And I’m almost positive I’m in the overwhelming majority. I’ll refrain from passing judgment on the foldable iPhone until it’s announced, but I can judge the Galaxy Z Fold 8 — and I think it exposes too many compromises with this device class. None of this is Samsung’s fault, of course, but it does a great job illustrating why foldable phones will probably never be ready for primetime. There was a time many years ago when I was bullish on the folding phone, hoping technology would progress far enough that they’d be equivalent in price and features to their non-folding counterparts. That time has not come yet, and I’m not confident it ever will. Samsung and Apple have tried addressing the biggest drawback to folding phones, the form factor, this year. But the Galaxy Z Fold 8 proves that it’s just not enough.