I recently wrote up some thoughts on the current hype around ML and AI. I sent it to the Risks Digest. Peter Neumann (the moderator) published a much-abbreviated version. This is the complete set of comments.


There is a massive miasma of hype and misinformation around topics related to AI, ML, and chat programs and how they might be used…or misused. I remember previous hype cycles around 5th-generation systems, robotics, and automatic language translation (as examples). The enthusiasm each time resulted in some advancements that weren’t as profound as predicted. That enthusiasm faded as limitations became apparent and new bright, shiny technologies appeared to be chased.

The current hype seems even more frantic for several reasons, not least of which is that there are many more potential market opportunities for recent developments. Perhaps the entities that see new AI systems as a way to reduce expenses by cutting headcount and replacing people with AI are one of the biggest drivers causing both enthusiasm and concern (see, for example, this article). That was a driver of the robotics craze some years back, too. The current cycle has already had an impact on some creative media, including being an issue of contention in the media writers' strike in the US. It also is raising serious questions in academia, politics, and the military.

There’s also the usual hype cycle FOMO (fear of missing out) and the urge to be among the early adopters, as well as those speculating about the most severe forms of misuse. That has led to all sorts of predictions of outlandish capabilities and dire doom scenarios — neither of which is likely wholly accurate. AI, generally, is still a developing field and will produce some real benefits over time. The limitations of today's systems may or may not be present in future systems. However, there are many caveats about the systems we have now and those that may be available soon that justify genuine concern.

First, LLMs such as ChatGPT, Bard, et al. are not really "intelligent." They are a form of statistical inference based on a massive ingest of data. That is why LLMs "hallucinate" -- they produce output that matches their statistical model, possibly with some limited policy shaping. They are not applying any form of "reasoning," as we define it. As noted in a footnote in my recent book,

Philosophically, we are not fond of the terms 'artificial intelligence' and 'machine learning,' either. Scholars do not have a good definition of intelligence and do not understand consciousness and learning. The terms have caught on as a shorthand for 'Developing algorithms and systems enhanced by repeated exposure to inputs to operate in a manner suggesting directed selection.' We fully admit that some systems seem brighter than, say, certain current members of Congress, but we would not label either as intelligent.

I recommend reading this and this for some other views on this topic. (And, of course, buy and read at least one copy of Cybermyths and Misconceptions.

Depending on the data used to build their models, LLMs and other ML systems may contain biases and produce outright falsehoods. There are many examples of this issue, which is not new: bias in chatbots (e.g., Microsoft Tay turning racist, bias in court sentencing recommendation systems, and bias in facial recognition systems such as discussed in the movie Coded Bias ). More recently, there have been reports showing racial, religious, and gender biases in versions of ChatGPT (as example, this story). “Hallucinations” of non-existent facts in chatbot output are well-known. Beyond biases and errors in chats, one can also find all sorts of horror stories about autonomous vehicles, including several resulting in deaths and serious injuries because they aren’t comprehensive enough for their uses.

These limitations are based on how the systems are trained. However, it is also possible to "poison" these systems on purpose by feeding them bad information or triggering the recall of biased information. This is an area of burgeoning study, especially within the security community. Given that encoded systems derived in these large ML models cannot be easily reversed to understand precisely what causes certain decisions to be made (often referred to as "explainable AI"), there are significant concerns about inserting these systems in critical paths.

Second, these systems are not accountable in current practice and law. If a machine learning system (I'll use that term but cf my 2nd paragraph) comes up with an action that results in harm, we do not have a clear path of accountability/responsibility. For instance, who should be held at fault if an autonomous vehicle were to run down a child? It is not an "accident" in the sense that it could not be anticipated. Do we assign responsibility to the owner of the vehicle? The programmers? The testers? The stockholders of the vendor? We cannot say that "no one" is responsible because that leaves us without recourse to force a fix of any underlying problems, of potential recompense to the victims, and to general awareness for the public. Suppose we use such systems safety or correctness-critical systems (and I would put voting, healthcare, law enforcement, and finance as exemplars). In that case, it will be tempting for parties to say, "The computer did it," rather than assign actual accountability. That is obviously unacceptable: We should not allow that to occur. The price of progress should not be to absolve everyone of poor decisions (or bad faith). So who do we blame?

Third, the inability of much of the general public to understand the limitations of current systems means that any use may introduce a bias into how people make their own decisions and choices. This could be random, or it could be manipulated; either way, it is dangerous. It could be anything from gentle marketing via recency effects and priming all the way to Newspeak and propaganda. The further towards propaganda we go, the worse the outcome may be. Who draws the line, and where is it drawn?

One argument is, "If you train humans on rampant misinformation, they would be completely biased as well, so how is this different?" Well, yes -- we see that regularly, which is why we have problems with Q-anon, vaccine deniers, and sovereign citizens (among other problem groups). They are social hazards that endanger all of us. We should seek ways to reduce misinformation rather than increase it. The propaganda that is out there now is only likely to get worse when chatbots and LLMs are put to work, producing biased and false information. This has already been seen (e.g., this story about deepfakes), and there is considerable concern about the harm this can bring. Democracy is intended to work best when the voters have access to accurate information. The rising use of these new generative AI systems is already raising the specter of more propaganda, including deep-fake videos.

Another problem with some generative systems (artwork, generating novels, programming) is that they are trained on information that might have restrictions, such as copyright. This raises some important questions about ownership, creativity, and our whole notion of issues of rule of law; the problems of correctness and accountability remain. There is some merit to the claim that systems trained on (for example) art by human artists may be copying some of that art in an unauthorized manner. That may seem silly to some technologists, but we’ve seen lawsuits successfully executed against music composers alleged to have heard a copyrighted tune at some point in the past. The point is the law (and perhaps more importantly, what is fair) is not yet conclusively decided in this realm.

And what of leakage? We’re already seeing cases where some LLM systems are ingesting the questions and materials people give them to generate output. This has resulted in sensitive and trade secret materials being taken into these databases…and possibly discoverable by others with the right prompting (e.g., this incident at Samsung). What of classified material? Law enforcement sensitive material? Material protected by health privacy laws? What happens for models that are used internationally when the laws are not uniform? Imagine the first “Right to be forgotten” lawsuits against data in LLMs. There are many questions yet to be decided, and it would be folly to assume that computing technologists have thoroughly explored these issues and designed around them.

As I wrote at the beginning, there are potential good uses for some of these systems, and what they are now is different from what they will be in, for example, a decade. However, the underlying problem is what I have been calling "The Trek futurists" -- they see all technology being used wisely to lead us to a future roughly like in Star Trek. However, humanity is full of venal, greedy, and sociopathic individuals who are more likely to use technology to lead us to a "Blade Runner" future ... or worse. And that is not considering the errors, misunderstandings, and limitations surrounding the technology (and known to RISKS readers). If we continue to focus on what the technology might enable instead of the reality of how it will be (mis)used, we are in for some tough times. One of the more recent examples of this general lack of technical foresight is cryptocurrencies. They were touted as leading to a more democratic and decentralized economy. However, some of the highest volumes of uses to date are money laundering, illicit marketplaces (narcotics, weapons, human trafficking, etc.), ransomware payments, financial fraud, and damage to the environment. What valid uses of cryptocurrency there might be (if there are any) seem heavily outweighed by the antisocial uses.

We should not dismiss, out of hand, warnings about new technologies and accuse those advocating caution as “Luddites.” Indeed, there are risks to not developing new technologies. However, the more significant risk may be assuming that only the well-intentioned will use them.

I have attended 14 of the last 22 RSA conferences. (I missed the last three because of COVID avoidance; many people I know who went became infected and contributed to making them superspreader events. I saw extremely few masks this year, so I will not be surprised to hear of another surge. I spent all my time on the floor and in crowds with a mask -- I hope that was sufficient.)

I have blogged here about previous iterations of the conference (2007, 2014, 2016, and most recently, 2019). Reading back over those accounts makes me realize that little has really changed. Some of the emphasis has changed, but most of what is exhibited and presented is not novel nor does it address root causes of our problems.

Each year, I treasure meeting with old friends and making some worthwhile new acquaintances with people who actually have a clue (or two). Sadly, the number of people I stop to chat with who don't have the vaguest idea about the fundamentals of the field or its history continue to constitute the majority. How can the field really progress if the technical people don't really have a clue what is actually known about security (as opposed to known about the products in their market segment)?

I was relieved to not see hype about blockchain (ugh!) or threat intelligence. Those were fads a few years ago. Apparently, hype around quantum and LLMs has not yet begun to build in this community. Zero trust and SBOM were also understated themes, thankfully. I did see more hardware-based security, some on OT, and a little more on user privacy. All were under-represented.

My comments on the 2019 RSAC could be used almost word-for-word here. Rather than do that, I strongly suggest you revisit those comments now.

Why did I go if I think it was so uninspiring? As usual, it was for people. Also, this year, I was on a panel for our recent book, Cybersecurity Myths and Misconceptions.. Obviously, I have a bias here, but I think the book addresses a lot of the problems I am noting with the conference. We had a good turnout at the panel session, which was good, but almost no one showed up at the book signings. I hope that isn't a sign that the book is being ignored, but considering it isn't hyping disaster or a particular set of products, perhaps that is what is happening. Thankfully, some of the more senior and knowledgable people in the field did come by for copies or to chat, so there is at least that. (I suggest that after you reread my 2019 comments, you get a copy of the book and think about addressing some of the real problems in the field.)

Will I go to the 2024 RSAC Conference? It depends on my health and whether I can find funds to cover the costs: It is expensive to attend, and academics don't have expense accounts. If I don't go, I will surely miss seeing some of the people who I've gotten to know and respect over the years. However, judging by how many made an effort to find me and how the industry seems to be going, I doubt will be missed if I am not there. That by itself may be enough reason to plan an alternate vacation

If you didn’t get a chance to attend S4x23 to hear the talks, or you simply haven’t heard enough from Spaf yet, here is a recording of the keynote interview with Spaf by Dale Peterson. The interview covered a lot of ground about the nature of defensive security, the new Cybermyths book (got yours yet?), OT security, the scope of security understanding, having too much information, and having a good security mindset.

This and other interviews and talks Spaf has given are on the Professor Spaf YouTube channel.

 

At the 25th anniversary CERIAS Symposium on March 29, we made a special awards presentation.

Unfortunately, I had lost my voice. Joel Rasmus read my remarks (included in what follows). I want to stress that these comments were heartfelt from all of us, especially me.

25 years ago, I agreed to start something new—something, unlike anything that had existed at Purdue before. I soon discovered that it was unlike any other academic center others had encountered: a multidisciplinary center built around the concept of increasing the security and safety of people by addressing problems from, and with, computing. I note that I wasn’t the only faculty member involved. Core faculty at the time were Sam Wagstaff,  Mike Atallah, and Carla Brodley, then in our School of ECE.  Sam and Mike have been steady contributors for more than 25 years (stretching back to the pre-CERIAS, COAST days); as an Emeritus Professor, Sam is still working with us.

I knew I needed help making the new entity succeed. My first step was hiring some great staff—Andra Nelson (now Martinez) and Steve Hare were the first two new hires; the late Marlene Walls was already working for me. Those three played a huge role in getting CERIAS running and helping with an initial strategic plan. We have recognized them in the past (and will feature them prominently in the history of CERIAS when I get around to writing it).

I quickly followed those hires by organizing an advisory board. Some of the members were personnel from the organizations that were committed to supporting us. Others were people in senior positions in various agencies and companies. And a few were friends who worked in related areas.

Those choices seem to have worked out pretty well. CERIAS grew from four involved faculty in April 1998 to (as of March 2023) 163. We went from four supporting companies and agencies to two dozen. We have thousands of alumni and worldwide recognition. There is considerable momentum for excellence and growth in the years to come.

CERIAS has benefited from the counsel, support, and leadership of scores of wonderful people from strategic partner organizations who served on the External Advisory Board over the years. However, some particularly stand out because they went above and beyond in their efforts to help CERIAS succeed. On this special occasion of our 25th anniversary,  we recognize six exceptional advisors who helped CERIAS succeed and be what it is today. 

(Unfortunately, due to various issues, none were present at the Symposium in person to receive the awards. This post is to share with everyone else how much we value their history with us.)

Silver Medals

We are bestowing five silver Foundation Award Medals to these individuals:

• Dr. Sidney Karin. Sid was a founder of the National Supercomputer Center program and was the founder and director of the Supercomputing Center at San Diego. He was a pioneer in that field and has received numerous recognitions for his leadership in supercomputing and networking. Sid graciously volunteered his time and tremendous expertise to sit on our advisory board for our formative years, providing insight into structuring and running an academic center.
• David Ladd.  David was (and is) with Microsoft, (then) working in university support and cybersecurity. He volunteered for our board and served as one of the rotating chairs. He also organized strong support from Microsoft, ensuring we had equipment, guest speakers, and internships for our students. He was a voice for Microsoft and industry, but more importantly, a strong voice for practical research.
• John Richardson.  John was with the Intel Corporation and an enthusiastic supporter of CERIAS. He also served as one of the rotating chairs as a member of the EAB. John went above and beyond to help secure guest speakers, equipment, student internships, and other companies’ support. He also put strong research and the welfare of the students ahead of his company’s interests.
• Dr.Robert E. (Bob) Roberts. Bob was the Chief Scientist for the Institute for Defense Analyses (IDA), an FFRDC well-known to those in government.  He provided great wisdom as a member of our EAB, including deep insights into understanding some conflicting requirements within the government. By training, he wasn’t a computer scientist, but his breadth of knowledge across many scientific disciplines helped us navigate many of our multidisciplinary issues.
• The late Emil Sarpa, the Manager of External Relations at Sun Microsystems. Emil did not serve on the board, but he was constantly present, ensuring that CERIAS had every computing resource we could need from Sun Microsystems, including many items in pre-release. He helped make introductions in the industry and got our students into fantastic opportunities. His support began pre-CERIAS with one of the initial grants that started the COAST Laboratory, and he ensured that Sun was CERIAS’s biggest founding partner.

These five people provided assistance above and beyond what we expected, and we will be forever grateful.

Gold Medal

We had one final, special award.

Timothy Grance has been a mainstay at NIST (National Institute for Standards and Technology) for decades. You can find his name on many of the reports and standards NIST has issued and other computing and cybersecurity activities. He’s not as well known as many of our advisors because he prefers to provide quiet, steady contributions. Most importantly to CERIAS, Tim has great vision and is one of the rare people who can find ways to help others work together to solve problems. He is inspirational, thoughtful, and cares deeply about the future. These qualities have undoubtedly been useful in his job at NIST, but he brought those same skills to work for CERIAS at Purdue and even before as an advisor to COAST.

For the last 25 years, Tim was (and continues to be) an honored member of the External Advisory Board. He has attended countless board meetings and events over the years — all at his personal expense. He made introductions for us across a wide variety of institutions—academic, governmental, and commercial—and hosted some of the EAB meetings. He has always provided sage advice, great direction, and quiet support for all we have done.  Despite being somewhat limited by a significant stroke a few years ago, he fought back courageously and returned to CERIAS for our Symposium and Board meeting. We reserve a chair for him even when he cannot travel to be with us. 

Tim’s commitment to the field, especially to CERIAS, make him a national treasure. We are proud also to consider him a CERIAS treasure, and thus award the Gold Foundation Award Medal to Timothy Grance.

Thank you

We conclude with sincere thanks, not only to these six wonderful people, but to all those who, over the years, have provided support, advice, time, equipment, funding, problem sets, and simply good cheer. That CERIAS has made it 25 years successfully and continues to grow and innovate is a testament to the importance of the problems and the willingness of such a large community to help address them. Time has only grown the problem set, but everyone associated with CERIAS is ready and willing to take them on. We all look forward to continuing our engagement with the community in doing so!

I recently saw an account of malicious compliance recounted in r/eddit and quoted in a Mastodon thread

Not allowed to work from home so I don't
My job recently told me that even during the snowstorm we got earlier this week, I am not allowed to work from home at all. Even though I work in IT and do everything remotely, they want me in the office.
So I deleted Teams and my email off my phone. I am no longer available after hours.
My boss tried to call me for something urgent last night and couldn't reach me. He asked why today and I explained to him what I was told.
I am not allowed to work from home.

It prompted me to think of several instances where I have engaged in behavior that might be described as malicious compliance; I prefer to think of them as instances of "security compliance education." Here's one such instance that my students see enjoy hearing about.

In 2000, we got some funding from a US federal agency (which will be unnamed) to explore for potential vulnerabilities in a commercial printer/copier combination. My technical point of contact (POC) told me that we didn't need to file any reports until we had some results. Apparently, he didn't convey this to the agency business person because the contract specified a long, convoluted monthly report. I was forcibly reminded of this requirement a week after the contract was finalized, even though it was in the midst of the winter break, and absolutely nothing had happened -- or would happen, for at least another month.

I grumbled a bit but compiled the report with basically "nothing to report" and "nothing spent" in the various sections and uploaded it via FTP to their designated site as a PDF.

Now, it is important to this story that my standard computers for use at the time were Sun workstations and Macintosh systems. Most of the research we did was on these systems, and our papers and reports were produced using LaTeX. We avoided Windows because it was usually so buggy (blue screens) and so prone to security problems. We also avoided Word because (a) it was (and is) annoying, and (b) it was a common vector for computer viruses. Thus, my monthly report was produced using LaTeX.

Two weeks into the semester, I got an email from some clerk at the sponsoring agency noting that the monthly report must be submitted as a Word document; the contract specified Word and only Word, and I must submit the report as a Word document, with no deviation allowed. I placed a call to my POC, and he indicated, apologetically, that he could not alter the terms as they were standard for the agency involved: everyone had to abide by them.

Grrrrr....

So, after a little thought,¹ I produced the next monthly report in LaTeX as before. I produced a PDF of the report and printed it. Then, I scanned each sheet individually into a graphic file (.pic, as I recall). I then rebooted one of our Windows machines² into MS-DOS and loaded up the oldest version of MS Word I could locate. After consulting the manual, I created a document where each page contained an image -- the corresponding image for that page of the report I had prepared. I saved it out to disk (it was huge), and uploaded it to the sponsor FTP site. Yes, it was basically a huge file of graphic images, but it was technically a Word file.

The next day I got an automated response noting the submission. Three days later, I got an email asking if the report was what I actually intended to upload. I responded that yes, it was. I indicated it had all the required information and was most definitely a Word document. I also alerted my POC about the upload (he was amused).

Another few days later and I got email from the original person who had complained about the PDF now complaining they were having difficulty with the file. I responded that the contract required Word, and that is what I used -- I wasn't responsible for their IT issues.

In month 3, I went through the same procedure but didn't have the email exchanges. Purdue then got an email from the agency business office stating that they were altering their standard business practices to allow all contractor reports to be submitted in Word -or- PDF. Would we mind submitting PDF henceforth? I briefly weighed the idea of continuing my production of Word versions of the report but decided that changing the business practices of a whole federal agency was enough.

---

Footnotes:
1. Someone once asked me why I didn't send them a Word document with some mischevious macros. I replied "USC 18 § 1030" (that's the Computer Fraud and Abuse Act).

2. Microsoft was a CERIAS partner at the time. When their rep visited, he saw that the lab was equipped with only Sun machines and Macintoshes. A few weeks later, we had several nice servers with Windows preinstalled delivered to the CERIAS lab. All our existing systems were named after mythical and fictional places (e.g., Amber, Oz, Dorsai, Uqbar), and we wanted to continue that scheme. We collectively decided to name the new machines Hel, Tartarus, and Niflheim. When he next visited and saw the machines, with nametags attached, he smiled a little. Two weeks later, we got another three, and they got related names; I can't recall exactly, but I think they were Underworld, Mictlan, and Jahannam). At his next visit, he remarked he could send us a lot more machines. I said we'd find a home for them, and welcome the chance to engage more of our philosophy, history, and literature faculty in the process.

All that said, we actually had a great working relationship with MS, and they hired a lot of our graduates. The machines did get a lot of use in experiments and classes.