O'Reilly on Data

JANUARY 16, 2024

ChatGPT was released just over a year ago (at the end of November 2022), and countless people have already written about their experiences using it in all sorts of settings. (I even contributed my own hot take last year with my O’Reilly Radar article Real-Real-World Programming with ChatGPT.) What more is left to say by now? Well, I bet very few of those people have actually chatted with ChatGPT.

Interactive 183

Interactive Recreation/Entertainment Visualization IT 183

O'Reilly on Data

JANUARY 9, 2024

Kevlin Henney and I recently discussed whether automated code generation, using some future version of GitHub Copilot or the like, could ever replace higher-level languages. Specifically, could ChatGPT N (for large N) quit the game of generating code in a high-level language like Python, and produce executable machine code directly, like compilers do today?

Modeling 280

Modeling Software Testing Optimization 280

O'Reilly on Data

DECEMBER 12, 2023

Generative AI stretches our current copyright law in unforeseen and uncomfortable ways. In the US, the Copyright Office has issued guidance stating that the output of image-generating AI isn’t copyrightable, unless human creativity has gone into the prompts that generated the output. This ruling in itself raises many questions: how much creativity is needed, and is that the same kind of creativity that an artist exercises with a paintbrush?

Modeling 251

Modeling Software Sales Statistics 251

O'Reilly on Data

NOVEMBER 30, 2023

Tim O’Reilly forwarded an excellent article about the OpenAI soap opera to me: Matt Levine’s “ Money Stuff: Who Controls Open AI.” I’ll skip most of it, but something caught my eye. Towards the end, Levine writes about Elon Musk’s version of Nick Bostrom’s AI that decides to turn the world to paper-clips : [Elon] Musk gave an example of an artificial intelligence that’s given the task of picking strawberries.

Technology 243

Technology IT 243

O'Reilly on Data

NOVEMBER 28, 2023

Generative AI has been the biggest technology story of 2023. Almost everybody’s played with ChatGPT, Stable Diffusion, GitHub Copilot, or Midjourney. A few have even tried out Bard or Claude, or run LLaMA 1 on their laptop. And everyone has opinions about how these language models and art generation programs are going to change the nature of work, usher in the singularity, or perhaps even doom the human race.

Enterprise 269

Enterprise Modeling Testing Reporting 269

O'Reilly on Data

NOVEMBER 14, 2023

This is not the first time that I’ve written about AI creativity, and I doubt that it will be the last. It’s a question that comes up repeatedly, and that is very much in the current mind, with events like the strikes by the Writers’ Guild of America and the Screen Actors Guild , in which the use of AI to create scripts and to generate images of actors was an issue.

Recreation/Entertainment 212

Recreation/Entertainment Marketing Publishing Data-driven 212

O'Reilly on Data

OCTOBER 31, 2023

This time of year, everyone publishes predictions. They’re fun, but I don’t find them a good source of insight into what’s happening in technology. Instead of predictions, I’d prefer to look at questions: what are the questions to which I’d like answers as 2023 draws to a close? What are the unknowns that will shape 2024? That’s what I’d really like to know.

Software 202

Software Modeling Technology Publishing 202

O'Reilly on Data

OCTOBER 30, 2023

Disclaimer: Based on the announcement of the EO , without having seen the full text. While I am heartened to hear that the Executive Order on AI uses the Defense Production Act to compel disclosure of various data from the development of large AI models, these disclosures do not go far enough. The EO seems to be requiring only data on the procedures and results of “Red Teaming” (i.e. adversarial testing to determine a model’s flaws and weak points), and not a wider range of inf

Measurement 296

Measurement Risk Modeling Metrics 296

O'Reilly on Data

OCTOBER 24, 2023

Ever since the current craze for AI-generated everything took hold, I’ve wondered: what will happen when the world is so full of AI-generated stuff (text, software, pictures, music) that our training sets for AI are dominated by content created by AI. We already see hints of that on GitHub: in February 2023, GitHub said that 46% of all the code checked in was written by Copilot.

Modeling 223

Modeling Statistics Software Reporting 223

O'Reilly on Data

OCTOBER 17, 2023

Anant Agarwal, an MIT professor and of the founders of the EdX educational platform, recently created a stir by saying that prompt engineering was the most important skill you could learn. And that you could learn the basics in two hours. Although I agree that designing good prompts for AI is an important skill, Agarwal overstates his case. But before discussing why, it’s important to think about what prompt engineering means.

Interactive 250

Interactive Software Testing IT 250

O'Reilly on Data

OCTOBER 10, 2023

Ethan and Lilach Mollick’s paper Assigning AI: Seven Approaches for Students with Prompts explores seven ways to use AI in teaching. (While this paper is eminently readable, there is a non-academic version in Ethan Mollick’s Substack.) The article describes seven roles that an AI bot like ChatGPT might play in the education process: Mentor, Tutor, Coach, Student, Teammate, Student, Simulator, and Tool.

Testing 182

Testing Modeling IT Risk 182

O'Reilly on Data

SEPTEMBER 19, 2023

I am wired to constantly ask “what’s next?” Sometimes, the answer is: “more of the same.” That came to mind when a friend raised a point about emerging technology’s fractal nature. Across one story arc, they said, we often see several structural evolutions —smaller-scale versions of that wider phenomenon. Cloud computing?

Machine Learning 245

Machine Learning Modeling Testing Cost-Benefit 245

O'Reilly on Data

SEPTEMBER 12, 2023

A few weeks ago, I saw a tweet that said “Writing code isn’t the problem. Controlling complexity is.” I wish I could remember who said that; I will be quoting it a lot in the future. That statement nicely summarizes what makes software development difficult. It’s not just memorizing the syntactic details of some programming language, or the many functions in some API, but understanding and managing the complexity of the problem you’re trying to solve.

Software 304

Software Management Modeling Metrics 304

O'Reilly on Data

AUGUST 15, 2023

To follow up on our previous survey about low-code and no-code tools, we decided to run another short survey about tools specifically for software developers—including, but not limited to, GitHub Copilot and ChatGPT. We’re interested in how “developer enablement” tools of all sorts are changing the workplace. Our survey 1 showed that while these tools increased productivity, they aren’t without their costs.

Software 182

Software Reporting Marketing Enterprise 182

O'Reilly on Data

AUGUST 8, 2023

I’m sure that nobody will be surprised that the number of searches for ChatGPT on the O’Reilly learning platform skyrocketed after its release in November, 2022. It might be a surprise how quickly it got to the top of our charts: it peaked in May as the 6th most common search query. Then it dropped almost as quickly: it dropped back to #8 in June, and fell further to #19 in July.

Machine Learning 182

Machine Learning Modeling Software IT 182

O'Reilly on Data

JULY 25, 2023

If you’re reading this, chances are you’ve played around with using AI tools like ChatGPT or GitHub Copilot to write code for you. Or even if you haven’t yet, then you’ve at least heard about these tools in your newsfeed over the past year. So far I’ve read a gazillion blog posts about people’s experiences with these AI coding assistance tools. These posts often recount someone trying ChatGPT or Copilot for the first time with a few simple prompts, seeing how it does for some small self-containe

Consulting 246

Consulting Interactive Software IT 246

O'Reilly on Data

JULY 18, 2023

Imagine for a minute that you’re a programming instructor who’s spent many hours making creative homework problems to introduce your students to the world of programming. One day, a colleague tells you about an AI tool called ChatGPT. To your surprise (and alarm), when you give it your homework problems, it solves most of them perfectly, maybe even better than you can!

Snapshot 154

Snapshot Modeling Strategy IT 154

O'Reilly on Data

JULY 11, 2023

There’s a lot of angst about software developers “losing their jobs” to AI, being replaced by a more intelligent version of ChatGPT, GitHub’s Copilot, Google’s Codey, or something similar. Matt Welsh has been talking and writing about the end of programming as such. He’s asking whether large language models eliminate programming as we know it, and he’s excited that the answer is “yes”: eventually, if not in the immediate future.

Testing 248

Testing Software Visualization Interactive 248

O'Reilly on Data

JUNE 27, 2023

Does your company plan to release an AI chatbot, similar to OpenAI’s ChatGPT or Google’s Bard? Doing so means giving the general public a freeform text box for interacting with your AI model. That doesn’t sound so bad, right? Here’s the catch: for every one of your users who has read a “Here’s how ChatGPT and Midjourney can do half of my job” article, there may be at least one who has read one offering “Here’s how to get AI chatbots to do som

Risk Management 223

Risk Management Risk Management Modeling 223

O'Reilly on Data

JUNE 20, 2023

Understanding AI’s mysterious “opaque box” is paramount to creating explainable AI. This can be simplified by considering that AI, like all other technology, has a supply chain. Knowing what makes up the supply chain is critical to enforcing the security of the AI system, establishing trust with the consumer of the AI’s output, and protecting your organization from undue risk.

Modeling 197

Modeling Machine Learning Enterprise Technology 197

O'Reilly on Data

JUNE 15, 2023

The world changed on November 30, 2022 as surely as it did on August 12, 1908 when the first Model T left the Ford assembly line. That was the date when OpenAI released ChatGPT, the day that AI emerged from research labs into an unsuspecting world. Within two months, ChatGPT had over a hundred million users—faster adoption than any technology in history.

Metrics 283

Metrics Reporting Measurement Finance 283

O'Reilly on Data

JUNE 15, 2023

“Mitigating the risk of extinction from A.I. should be a global priority alongside other societal-scale risks, such as pandemics and nuclear war,” according to a statement signed by more than 350 business and technical leaders, including the developers of today’s most important AI platforms. Among the possible risks leading to that outcome is what is known as “ the alignment problem.

Risk 244

Risk Optimization Modeling Reporting 244

O'Reilly on Data

JUNE 13, 2023

A few months ago, I wrote about some experiments with prime numbers. I generated a 16-digit non-prime number by multiplying two 8-digit prime numbers, and asked ChatGPT (using GPT -3.5) whether the larger number was prime. It answered correctly that the number was non-prime, but when it told me the number’s prime factors, it was clearly wrong. It also generated a short program that implemented the widely used Miller-Rabin primality test.

Testing 208

Testing Publishing IT Risk 208

O'Reilly on Data

MAY 9, 2023

It’s hard to ignore the discussion around the Open Letter arguing for a pause in the development of advanced AI systems. Are they dangerous? Will they destroy humanity? Will they condemn all but a few of us to boring, impoverished lives? If these are indeed the dangers we face, pausing AI development for six months is certainly a weak and ineffective preventive.

Modeling 212

Modeling Reporting Data Governance Software 212

O'Reilly on Data

APRIL 25, 2023

This post is a brief commentary on Martin Fowler’s post, An Example of LLM Prompting for Programming. If all I do is get you to read that post, I’ve done my job. So go ahead–click the link, and come back here if you want. There’s a lot of excitement about how the GPT models and their successors will change programming. That excitement is merited. But what’s also clear is that the process of programming doesn’t become “ChatGPT, please build me an enterprise application to sell shoes.

Testing 325

Testing Software Strategy Enterprise 325

O'Reilly on Data

APRIL 5, 2023

The notion that artificial intelligence will help us prepare for the world of tomorrow is woven into our collective fantasies. Based on what we’ve seen so far, however, AI seems much more capable of replaying the past than predicting the future. That’s because AI algorithms are trained on data. By its very nature, data is an artifact of something that happened in the past.

Cost-Benefit 274

Cost-Benefit Data-driven Measurement Visualization 274

O'Reilly on Data

MARCH 23, 2023

ChatGPT, or something built on ChatGPT, or something that’s like ChatGPT, has been in the news almost constantly since ChatGPT was opened to the public in November 2022. What is it, how does it work, what can it do, and what are the risks of using it? A quick scan of the web will show you lots of things that ChatGPT can do. Many of these are unsurprising: you can ask it to write a letter, you can ask it to make up a story, you can ask it to write descriptive entries for products in a catalog.

IT 260

IT Modeling Testing Risk 260

O'Reilly on Data

MARCH 14, 2023

A couple of days ago, I was thinking about what you needed to know to use ChatGPT (or Bing/Sydney, or any similar service). It’s easy to ask it questions, but we all know that these large language models frequently generate false answers. Which raises the question: If I ask ChatGPT something, how much do I need to know to determine whether the answer is correct?

Testing 260

Testing Modeling IT 260

O'Reilly on Data

FEBRUARY 16, 2023

It’s been well publicized that Google’s Bard made some factual errors when it was demoed, and Google paid for these mistakes with a significant drop in their stock price. What didn’t receive as much news coverage (though in the last few days, it’s been well discussed online) are the many mistakes that Microsoft’s new search engine, Sydney, made.

Testing 210

Testing Statistics Modeling Optimization 210

O'Reilly on Data

FEBRUARY 14, 2023

Everybody knows about ChatGPT. And everybody knows about ChatGPT’s propensity to “make up” facts and details when it needs to, a phenomenon that’s come to be called “hallucination.” And everyone has seen arguments that this will bring about the end of civilization as we know it. I’m not going to argue with any of that. None of us want to drown in masses of “fake news,” generated at scale by AI bots that are funded by organizations whose intentions are most likely malign.

Optimization 240

Optimization Modeling Software IT 240

O'Reilly on Data

JANUARY 17, 2023

What would you say is the job of a software developer? A layperson, an entry-level developer, or even someone who hires developers will tell you that job is to … well … write software. Pretty simple. An experienced practitioner will tell you something very different. They’d say that the job involves writing some software, sure. But deep down it’s about the purpose of software.

Machine Learning 325

Machine Learning Predictive Modeling Software Modeling 325

O'Reilly on Data

JANUARY 10, 2023

The holidays are over, and it’s time to digest. Although I don’t subscribe to the idea that history or technology moves in jerky one-year increments, it’s still valuable to take stock, look at what’s happened, and decide what was important and what isn’t. We started the year with many people talking about an “AI winter.” A quick Google search will show that anxiety about an end to AI funding has continued through the year.

Interactive 248

Interactive Modeling Technology Enterprise 248

O'Reilly on Data

NOVEMBER 29, 2022

Major catastrophes can transform industries and cultures. The Johnstown Flood, the sinking of the Titanic, the explosion of the Hindenburg, the flawed response to Hurricane Katrina–each had a lasting impact. Even when catastrophes don’t kill large numbers of people, they often change how we think and behave. The financial collapse of 2008 led to tighter regulation of banks and financial institutions.

Machine Learning 277

Machine Learning Recreation/Entertainment Risk Testing 277

O'Reilly on Data

OCTOBER 18, 2022

A recently passed law in New York City requires audits for bias in AI-based hiring systems. And for good reason. AI systems fail frequently, and bias is often to blame. A recent sampling of headlines features sociological bias in generated images , a chatbot , and a virtual rapper. These examples of denigration and stereotyping are troubling and harmful, but what happens when the same types of systems are used in more sensitive applications?

Risk Management 291

Risk Management Risk Testing Measurement 291

O'Reilly on Data

OCTOBER 11, 2022

Gartner has anointed “Hyperautomation” one of the top 10 trends for 2022. Should it be? Is it a real trend, or just a collection of buzzwords? As a trend, it’s not performing well on Google; it shows little long-term growth, if any, and gets nowhere near as many searches as terms like “Observability” and “Generative Adversarial Networks.” And it’s never bubbled up far enough into our consciousness to make it into our monthly Trends piece.

Data Integration 289

Data Integration Insurance Dashboards Modeling 289

O'Reilly on Data

AUGUST 9, 2022

In a previous article , I wrote about how models like DALL-E and Imagen disassociate ideas from technique. In the past, if you had a good idea in any field, you could only realize that idea if you had the craftsmanship and technique to back it up. With DALL-E, that’s no longer true. You can say, “Make me a picture of a lion attacking a horse,” and it will happily generate one.

Software 263

Software Testing Modeling IT 263

O'Reilly on Data

AUGUST 4, 2022

Humans are notoriously poor at judging distances. There’s a tendency to underestimate, whether it’s the distance along a straight road with a clear run to the horizon or the distance across a valley. When ascending toward a summit, estimation is further confounded by false summits. What you thought was your goal and end point turns out to be a lower peak or simply a contour that, from lower down, looked like a peak.

Machine Learning 260

Machine Learning Modeling Statistics Software 260