Tim McConnaughy: 0:14

Hello and welcome back to another episode of the Cables to Cloud podcast. I am your host this week, tim, and with me, as always, is my co-host, chris Kilometers. He's living in a land down under.

Chris Miles: 0:26

Officially changed the name. Yeah.

Tim McConnaughy: 0:29

And with us we have a very special returning guest, a good friend of us, good friend of the podcast, duan Lightfoot. How are you doing, duan?

Du'An Lightfoot: 0:36

Hey, what's up Tim, what's up Chris? And everyone out there in podcast land thanks for having me on.

Tim McConnaughy: 0:46

Excellent. So, just in case you know, we have to just real quick, because of course we don't know if people missed the last episode that we had you on Just do a real quick intro yourself and we'll just roll right into the topic today.

Du'An Lightfoot: 0:53

Yeah, my name is DeJuan Lightfoot. I'm a developer advocate at AWS. I'm not representing AWS, but I am speaking my views of what I've learned over my 20 year career in tech, from being a system administrator, network engineer, developer advocate, infrastructure automation engineer. I guess I'm a kind of generalist to learn wherever the trend takes me.

Tim McConnaughy: 1:15

Yep, and you're in. Your big thing now actually is you're like all in on the AI stuff, aren't you?

Du'An Lightfoot: 1:21

yes, yes, for the last, I guess um chat gbt came out in 2022 and I've been using it ever since january 2022, that is oh wow, yeah, yeah, right, the, the crossing, the chasm guy, you're like the, the first guy I'm. I'm definitely not the first guy, because you know, um gpt2 came out, I want to say like 2019 or 2020, something like that. So people are doing vi for a while now got you.

Tim McConnaughy: 1:47

So yeah, but anyway. So dewan you, we've invited you back. I mean, well, first of all, we just love having you on the podcast, so I appreciate you making the time, uh, but the other thing is that you fairly recently, within the last few months, uh attended autocon, which uh is an automation focus convention. Uh, is it? Is it caught? Is? Is it Denver where they have it? I'm trying to remember. Is it Colorado? Somewhere in Colorado, right?

Du'An Lightfoot: 2:10

Yeah, it was in Denver, Colorado. Autocon was a cool conference. It was different than most conferences I've been to. It was a smaller conference, had one track and it was really focused on being highly engaged in actual network automation For those that are in the community actually building things, writing scripts, deploying infrastructure configuration with code. They were all there. Well, about 500 people were there from the industry speaking on topics connecting building. I mean it was a good time.

Tim McConnaughy: 2:41

That's cool when you say it's small.

Chris Miles: 2:43

Oh yeah, sorry, go ahead. I was going to say yeah, yeah, I think I was talking to a friend of the pod, will collins, uh, after he had just got back from there and he said it was. He said it was the best conference he's ever been to, um, and he, says like just because it was, I think, just because it was kept relatively small and more contained, and you can have these like he said the hallway conversations were like phenomenal in that environment.

Du'An Lightfoot: 3:05

So yeah, yeah, it's really intimate, right. You go to a lot of conferences and they can kind of be overwhelming. So many people, so many things to do. You won't see everybody. This one you really get a chance to hang out with people that you've seen online, people that you know throughout your career that may have gone and they're in different companies. I met all types of people from all different walks of life and it was a really good experience.

Tim McConnaughy: 3:26

So I mean how. I mean how, when you say intimate convention, like paint a picture for the audience Like what are we talking? Like 500 people, like 300 people.

Du'An Lightfoot: 3:35

Yeah, it was 500 people in a single hotel, right A conference room in a hotel. So whenever the keynotes would happen or talks would happen since it was only one track everybody would converse in the same room, listen to the talk and then we all take a break together. And, yeah, you pretty much had to meet all the people you probably wanted to meet at the conference.

Tim McConnaughy: 3:55

That's cool. Yeah, I don't think there's enough of. I mean, there's like little regional events, I think, all over the country at any given time, but's it's hard to hit that, um, that uh sizing problem. Like you know, once it gets too big, you know, of course everybody goes to it, it's really popular, you got sponsors, they can make it big, but you definitely lose a lot of the uh, the intimacy. And then you know if it's too small, of course, then like nobody, nobody makes the trip, nobody shows up. You know, like it's it's a hard, it's a hard target to hit, right.

Du'An Lightfoot: 4:24

Yeah, normally when I leave a conference I'm tired, like I come back and I need some downtime. This one I was still recharged and ready to get back to work, so it was really good.

Tim McConnaughy: 4:34

That's awesome and so, like the whole convention obviously is focused on on some form of automation. Now, it's not. It's not just network automation, right, or is it mostly focused on network automation?

Du'An Lightfoot: 4:44

network automation right, or is it mostly focused on network automation? Yeah, it's mostly focused on network automation. Now there were some talks that spoke about um kind of networking in the industry, where we stand and how it applies to network automation, but the conference itself was pretty much around network automation yeah, so actually I get this is a good question then.

Tim McConnaughy: 5:00

so, given that you're you're deep in the ai trenches now and you went to a network automation convention, I mean, and and it's been for a few years for me since I've really played around with network automation and the crew that's been kind of pushing network automation, are you know, for the people there and maybe it's a biased, maybe it's a biased group, right, but are there still people thinking like, oh you know, automation is going to take my network engineering job?

Du'An Lightfoot: 5:29

Or is it you think it's finally starting to come around where people are really doing it? Yeah, this group is different, right? Because this group is actually building things, so you will have to talk to people that probably weren't at the conference, that's what I was thinking, too, when I was thinking of asking the question.

Tim McConnaughy: 5:38

I'm like ah, it's kind of a self-selection bias there, isn't it?

Chris Miles: 5:41

Yeah, if you're thinking ai is going to take your job, you're probably not attending the ai conference, right?

Du'An Lightfoot: 5:47

yeah, the ai conversation is different and we I guess we can kind of start talking about ai now. Yeah, the way I look at it, right, it's been an evolution. Um, network automation has been happening before the python hype. Yep, um, network engineers have been um using scripts, whether they're EEM scripts or whether they're Perl scripts, whether there's Bash scripts, some type of scripting, powershell, you name it. We've been doing network automation for years.

Du'An Lightfoot: 6:14

The thing that I see has kind of evolved is for one, containerization.

Du'An Lightfoot: 6:19

So, rather than running the code on your laptop, now we're able to centralize it and run it somewhere right and then we're able to automate it and schedule it and have it kick off, and then we're actually able to integrate APIs.

Du'An Lightfoot: 6:32

So now we can have these agents or workers that go out and grab information or data from our devices, load them into our database and then leverage an API to grab that data and display it to all the engineers on the team.

Du'An Lightfoot: 6:45

And all the engineers do not have to actually physically log into every device, they just talk to some web interface or some API, right?

Du'An Lightfoot: 6:52

So when we started talking about AI, I look at the next evolution is OK, I built all these tools to get interface to get routes, to add routes, to do all of these things in the network, to get interface to get routes, to add routes, to do all of these things in the network. But rather than me having to put these scripts together, can I use AI to kind of make decisions on what happened in my network? We're getting deep down that rabbit hole when it comes to agents, but I just think, when it comes to AI, since large language models like ChatGBT understand language, now they can kind of understand what's happening in our network and help us make more intelligent decisions and also free up time so we can do more higher end, complex tasks in the network yeah, I think I recall um a specific slide in your presentation that you had at autocon, um specifically kind of laying, kind of setting the scene a little bit about you know, around the AI hype, what is myth versus reality.

Chris Miles: 7:47

So could you take a couple minutes and kind of touch on what you think are the major myths versus realities that we're faced with with these network agents?

Du'An Lightfoot: 7:55

Yeah, I think, first, that AI will replace network engineers. I think that's a myth when you look at, let's say, the term AI agents, what? What that is is a system. You have the LLM as the brain of that system, but the actual agent is the code and the tools that network engineers are already building. So we still have to understand the network we and more, better in the future but we still need network engineers that understand the network, understand the infrastructure, to be able to think the models. When you give it a tool, let's say get interface, well, how does it know how to reason about that? Well, somebody needs to write the descriptions and help the model be able to understand the steps in the infrastructure, to know what tools are available for it to be able to get interface or to make a change or take a series of actions on your network. So no, it will not replace network engineers anytime soon, could?

Du'An Lightfoot: 8:54

it help us at tier one. Yes, it may could do that right. Tier two possibly could do some things there, but when we started looking at the architecture deep in the weeds networking we still need network engineers.

Tim McConnaughy: 9:06

Yeah, I mean like until AI can truly interpret a diagram and they're getting it's getting there right. I've seen some diagrams where it can understand like, hey, this line indicates that these two devices are linked together, and like that means something you know it kind of reasoned through that. I've seen some of that. Like that means something you know, kind of reasoned through that. I've seen some of that.

Tim McConnaughy: 9:24

But yeah, so on the agents, actually the agentic side of it, do you think that you know the network engineers of the future, that kind of like, are going to be using AI almost like a well, just as another tooling set?

Tim McConnaughy: 9:35

Really, do you think it's going to just okay, trying to think of how to say this? Do you think it's going to be more like people are going to go to chat, gpt or cloud or whatever and be like, hey, I'm a network engineer, I need to gather something from you know some Git interface or you know whatever from a thousand devices. You know what's the best way to do that and then like work with the agent to create the tooling for that. Or do you think it's going to be more like and maybe it's a little common to make L&B right, or do you think it's going to be more like you're going to build the agent, put it all together, build in the agentic kind of reasoning and all of that to use multiple tools, and then just kind of oh okay, and just walk away and kick it off.

Du'An Lightfoot: 10:16

Yeah, I think the first part. A lot of companies are already using some type of large language model for RAG retrieval, augmented generation. I have these wikis, I have this documentation, I have these things about my network and these guys to help engineers. Now can you retrieve it and help answer questions for my team? Right, they have some sort of simple system like that. So I think we're kind of already there at that point. Now to your point of using agents for our workflows.

Du'An Lightfoot: 10:44

I wonder if customizing agents per organization, how that will work, because when you think of network automation, that's hard right For a lot of teams. So if you layer AI on that, that introduces another level of complexity, so that may take some time. But will it help us write code? Sure, it's already doing that. Will it help us retrieve information and really understand our network a little better? Yes, it's already doing that. Will it help us analyze routes or ACLs? Yes, it's already doing that. So can it help us actually write configuration? Yes, it's already doing that. So it's doing a lot of these pieces.

Du'An Lightfoot: 11:23

For the question that you asked, now what would a real solution look like being implemented in an environment? I think we're going to get there gradually. We may have a tool that actually comes out the end of the year where everybody's like, yeah, this is it. This is what I see Right, because if I look at like Netbox, you already have your network state, they have all this Right, right. You can build an agent on top of that fairly easy. That one agent is not going to be the same for every environment, so to build a solution that works in every environment that's going to be kind of hard you know, because it had John Capobianco on here recently just kind of giving us kind of a breakdown on what agents are and how they work.

Chris Miles: 12:07

And you know, obviously the agent needs to be given a set of tooling in order to you know, use and find out information about the network that we're talking about. Right, and you know, like you said, it sounds like some of this tooling might be common things that engineers and architects are already using on a daily basis, which is cool, but it seems I mean, from my perspective, it sounded like a lot of that has to be API based.

Chris Miles: 12:32

And you know, I think the network realm has not been the one, that is, you know, leading the pack in terms of the best API support across the board.

Du'An Lightfoot: 12:42

Let's pause there. I don't think it's necessarily API-based right.

Du'An Lightfoot: 12:46

I think it's more so when you think of a tool, think of a. Let's take away the API. Just think Python function. Right, let's say you write some Python code that adds one plus two or multiplies three plus four, where you have two integers that are being passed into the function and then some arithmetic is happening within that function and then the solution is returned Right. Well, that's actually a function for an agent. You can have a math agent that solves math problems, right, there's no API involved in that. So if you have an infrastructure where you're making SSH or you're using something like NetMeco NetMeco, yeah, right. As long as your data is using something like Pydanic, where it's actually structured, you can actually do that. You don't necessarily need an API, you know. You just need a tool that takes inputs. You have Python function, because the actual agent is the system that handles the.

Du'An Lightfoot: 13:35

Let's take a step back. An LLM out the box, it can answer a question and return a response, but to get the weather, to check the date, to check your interfaces, it needs some type of function or tool that it knows about. Now it can't call that tool, but it can say hey, in my configuration I know, I have this tool and I know how to use it. So I asked the question get interfaces? It's going to say, okay, I know I have a tool. I can't tell you what the interface status is, but I know I need to use this tool. Let me reply back Tool usage. I need to use this tool.

Du'An Lightfoot: 14:09

And now, within your Python code, this is where the agentic workflow actually happens, because in your Python code or some type of orchestration tool, you actually handle that response from the agent to say, ok, the agent needs a tool. Now let me kick off this code, go get interfaces, get the results and reply back with a user message to the agent or the LLM to say, okay, here's the response, the tool result from the interfaces. Now the LLM will analyze that result and say, oh, okay, now I can answer your question based on these results.

Tim McConnaughy: 14:41

It's like an expansion on RAG, these results. It's like a. It's like an expansion on rag, like you know, except that it's like up to date, like truly you've run a tool, like you said, and actually fed that result of the rag kind of back into the llm, to the only difference the only, the only difference from like the way rag is implemented normally, or let's say traditional rag, right, rag happens with before it goes to the llm, right, so that's right before it goes to the llm.

Du'An Lightfoot: 15:08

When it comes to agents or tool usage, you actually send your initial prompt to the llm and the llm is able to reason or decide what tool to use and then it will start and let you know. Here's the tool I need to use, and your Python code needs to be able to handle that response, to be able to get the data or the information that the model needs to then be able to give you an accurate response.

Tim McConnaughy: 15:38

So it's like half and half right. Whatever the reply is or that you get back from the LLM you're writing handle like handling code that'll say oh well, I'll do this based on what you're telling me you need to do. Basically.

Du'An Lightfoot: 16:06

Right, and that's why, when we talk about implementing this into, do you have to troubleshoot your network automation? But you also have to troubleshoot the track and trace or the tracing or logging within those calls to your LLM, because the LLM you had to monitor the request going into the model, right, the prompt going into the model, the response is coming back from the model and then the orchestration that's happening in your agent. That's like four different steps or workflows is where you are at the log and troubleshoot and anything can go wrong at any given point.

Tim McConnaughy: 16:31

It's like a pipeline kind of I mean there's a lot more you know.

Chris Miles: 16:34

Yep, it is yeah it's like you just need to service chain all this stuff together to make sure it happens in the right operation. Right, and there's tools for that right.

Du'An Lightfoot: 16:43

So LaneChain provides a library for that. There's several others I know. On AWS we have Amazon Bedrock Agents. That kind of helps you orchestrate and we handle that for you. So a lot of that logging the, building out the agent functions or tools you can actually do that with Lambda functions.

Tim McConnaughy: 17:02

I was going to say it's.

Du'An Lightfoot: 17:02

Lambda right To your agents.

Tim McConnaughy: 17:09

Actually, I actually do that with lambda functions and I'm gonna say it's lambda, right to your agents. Actually, I haven't seen this, so I'm kind of curious now. So the front end for bedrock agents does it actually take you to like to lambda, or is it just like a wrapper around?

Du'An Lightfoot: 17:14

like a gooey wrapper. Okay, right, so you actually go into lambda. When you actually create your own functions, that the tools for your agent, you actually go into lambda, put your python code there and then you actually build this. You can build it out on a console or you can actually use I think it's JSON schema, don't get me wrong but to actually tell your agent what the objects or the tools or the functions that it has. You can do this in a console. But there's also a schema that we provide that you can actually update and give it to your agent so that you don't have to go in the console and click all around.

Chris Miles: 17:45

Right. One question that I do have is it's like a prevalent issue with AI that I think is partially being, you know, handled over the last few months, but there's still probably a lot of people out there that are like, oh, I don't want to use AI because I don't trust it, because hallucinations happen and I just can't trust the data that it provides to me. Right? So from your perspective, in this scenario where maybe you are doing things where it's giving you information about your network, what kind of verification process comes into that, and how do you avoid, or at least do some fact checking, with things like hallucinations?

Tim McConnaughy: 18:25

yeah, before pushing like big major. You know what is it?

Du'An Lightfoot: 18:30

uh, automation lets you fuck up at scale, right um, so hallucinations aren't a bad thing, and the reason why I say that is because if you look at a large language model, the way they're trained, they're trained on all the data of the internet. That's called pre-training, so the data is clean, sanitized, put together in documents and then you train on these documents and then you have a model that can spit out some document based that looks like it, looks like some words, looks like some language because it understands it, or labelers that basically ask questions that we all ask. They write these questions and then they write responses that the model should give, and so they do this for a ton of different questions, right? And then, once they have these questions, then the model is trained again to be a helpful assistant, right?

Du'An Lightfoot: 19:20

Interesting and so it has all of this information where it says, ok, I'm able to. I have all this data, I understand the questions that are asked and I'm able to make predictions to help kind to to get some type of accurate answers that I'm confident about Because, due to the mathematical, mathematical probability in the model, I'm able to predict the next token in the response. Probability in the model. I'm able to predict the next token in the response. Right, because it uses tokens, because it doesn't actually understand text and it understands tokens or embeddings or whatnot. When this happens, the hallucinations they are the model kind of guessing on how to respond. Now, sometimes they do get things wrong. We don't really understand that. But ways to check against that one guard rails. You put guard rails around a model. Those are it's like a firewall that goes around the model. You check the inputs, you check the outputs, and so you kind of do that around the model when it responds.

Du'An Lightfoot: 20:17

We have something on AWS called automated reasoning and so if you're looking for, let's say, a response from the model and you are able to determine let's say You're looking for IP addresses or you're looking for something that's structured and something that you can actually test against Well, you can use automated reasoning to ensure that the response is what you expect and the reason why the model responded this way. Also, you can actually parse your responses into the model and out of the model's text. Right, ok, so you can actually write tools to troubleshoot and test against the responses before it even returns your user, because you're still using code at the end of the day, right, and when it, when, when those, when, those solutions happen, you should log those responses so you can go back and see, okay, what was the question to ask. When we're writing questions to the model, there's something called prompt engineering. I'm sure you all heard of it, right?

Tim McConnaughy: 21:08

yeah.

Du'An Lightfoot: 21:08

Yeah, of course you can ask a basic question, but if that basic question doesn't contain accurate responses I mean accurate um content it's not posed correctly, it doesn't have enough information, it's not direct, there's a lot of fluff in it. It wasn't, it wasn't able to use rag correctly. There's a lot of things that could go wrong in that. So when you say a hallucination, we need to find out what actually happened with that prompt. And then, does it align with the response? Right, right. And so when we're building our applications because applications are being built what you're trying to do is evaluate the response before it gets to your end user. Okay, right.

Du'An Lightfoot: 21:47

So you may not have one model, you may have a collection of models, yeah, that are testing and validating responses before they even get to the user. Right, yeah. And then one more thing You're actually going to evaluate your models before you put those in production. So if you have, let's say, an agent or some type of workflow that's built now, you're going to want to test this 100 times, 200 times, to see how the prompts, or more, to see how the prompts are being responded. What is the accuracy? You're going to test this. This is just like you test any other code. You're going to test this as well.

Tim McConnaughy: 22:20

Because you mentioned guardrails and validation, how could you give me, give us an example of what, a how you would build a guardrail and like what that like? Are you just writing as part of your prompt a guardrail like validate? This response meets this criteria, or like I'm?

Du'An Lightfoot: 22:36

I'm struggling with the visualizing that yeah, so picture a large language model, picture a box around a large language model. Protects it, what goes in it, and protects what comes out of it, just like a firewall, right? And so if you were, let's use amazon bear, bear Rock, amazon Bear Rock, converse API or any other APIs you can actually create a guardrail. And let's say this guardrail is supposed to look for usernames, passwords, phone numbers, account numbers, credit cards, like.

Tim McConnaughy: 23:03

PII stuff.

Du'An Lightfoot: 23:04

PII stuff. Right, You're going to inspect the prompt to ensure that no PII is going into the model. If it does, you can tell it to match those numbers right and then, when it comes out from the model, you can tell it to match those numbers, or what you could do. Let's say someone has to ask a question that you don't want it to ask. Maybe they want to ask about sports betting or something that the model should not be answering. Well, before the question even goes to the model, guardrails can check the question and ensure that that question is blocked before it even goes to the model, which is going to help save on cost.

Tim McConnaughy: 23:38

Right, because you're not running the.

Chris Miles: 23:39

you're not running the query, you're not lighting up the model, the LLM also are tied to a, you know, a confidence score. Right, a scoring mechanism to say how likely the likelihood that the response is accurate, right? Can the guardrail kind of come into play on that kind of outward direction and say like, don't provide responses that are below a certain confidence score? Is that where you'd use a guardrail, or would that be somewhere else?

Du'An Lightfoot: 24:08

You can do that and I believe there are tools for that. There are tools for that on amazon bedrock. That, specifically I'm. I would have to get back to you on that if we have it. But yes, you can do that. Um, but what are you scoring against?

Tim McConnaughy: 24:22

I think I read a blog, I think we read a blog about that. Actually, uh, this was months ago, but I think I read that that was actually a new feature that they had added to bedrock actually was the confidence score. Yeah, uh, ability, actually I think it is. I didn't mean to hit you on the, I didn't mean to ask you. You know, like hit you in the gut, uh out of nowhere, but uh, because that's a. We certainly kind of meandered there a little bit so you probably weren't ready, but I do. I do sort of remember reading an amazon blog about, about bedrock adding uh, confidence score or something, if I remember correctly.

Du'An Lightfoot: 24:52

So there's a name for it, and we call it contextual grounding.

Tim McConnaughy: 24:55

That's it. That was it. Yeah, that's what it's called.

Du'An Lightfoot: 24:57

So when you configure a guardrail, you can check. There's like several different things. You can check for content filter, deny topics, sensitive information, word filters, contextual grounding. That's why I wasn't clear on it, because it's called contextual grounding and that's just a checkbox that you check when you configure it. You just turn it on and it actually does it.

Tim McConnaughy: 25:16

Okay, okay. So that's the type of check you would run if you were building, say, an agent that's gonna kick off some tooling to like go make changes on your network or something like that.

Chris Miles: 25:27

Is that accurate? Well, my question was just to make sure that is even something that you would put into a guardrail, or does that come in somewhere else in the kind of sequence?

Du'An Lightfoot: 25:35

But yes, that's doable via the guardrail. Yep, it's in the guardrail.

Tim McConnaughy: 25:38

I mean, but what you said is accurate, right, if you build a tool, and building an AI tool is the same as building any other code or whatnot. It's just you're talking to an LLM instead of writing Python code directly or something right? So you would have to run it. The part I'm still unclear about and this is because I just haven't done it is, let's say I built an agent and I wanted the agent to do X, y, z, it doesn't really matter whatever that workflow looks like and I run it and it'd say it doesn't do the workflow correctly, like how would you troubleshoot that? Would you go back to the prompt? Troubleshoot that? Would you go back to the prompt? Like I mean, let's just approve, let's just say that it's spat out nonsense, like ip addresses. Then you assume, like the ip, the tooling for handling the ip is wrong. I'm sorry, I didn't mean to nail you down on this, man, I'm just.

Du'An Lightfoot: 26:19

I'm just curious, because we kind of you've now activated my trap card, I guess, because now you got me thinking so, just like, python code is involved, right, so you python code is going to be involved, or some type of code, right, you can use javascript, you can use whatever, but let's just stick with python because that's what most networking is used. When it comes to what you're talking about, you're going to have logging and with every application you build, there's going to be some type of logging. Now, when we're communicating with the llm, in order for your llm, which is the brain of your agent, it's, it's not right. The, the llm itself is not the agent.

Du'An Lightfoot: 26:54

The orchestration that you have is what makes the agent. When you actually build out your API calls to your LLM. There's two things you need. You need a system message. The system messages tell the agent the role that it has. Right, you're a network engineer, the tools that it has right, right, the arguments that those tools accept, right. So let's say, your, your agent, is a weatherman, right? Or weather woman? Well, that agent is going to need, is going to need, to know that it's, so it has access to the weather, right?

Du'An Lightfoot: 27:26

right and access to get weather. Also need to know what cities or states or locations right, so you need to-.

Tim McConnaughy: 27:32

Right. What are the inputs?

Du'An Lightfoot: 27:33

What are the inputs right? So you're going to talk to your agent and tell it in the system message all of these things so it can really be able to reason and understand when to use a tool and when not to use a tool. And what parameters do I need when I need this tool? Because I may need to ask the user okay, you want me to get the interface? What is the IP address? So in the system message you give it all of that information, right, and then from there the prompt is actually the user inputting their questions that they have Right, right. But the system message is what's going to be the, I would say, the secret sauce to making your agent have the understanding that it needs, or your LLM having the understanding it needs for the tools and functions that it has, the capabilities to.

Tim McConnaughy: 28:21

Yeah, it's like a preloaded prompt or like a role definition, like kind of that the user doesn't have to do right, the user only has to ask the question. But you've already, I see.

Du'An Lightfoot: 28:29

So we're troubleshooting right. You're logging the prompts that the users are um sending to the model. Well, since you're logging those prompts, what you could actually do is take one of the prompts, send it to your agent and see how it responds right your application to see what's being called, what's not being called. Did it use the tool? Did it not use the tool? What's the data coming back from the tool? And then, how did it process and respond once it had the data from the tool?

Tim McConnaughy: 28:54

Yeah, perfect.

Du'An Lightfoot: 28:55

Chances are, is that it could be something wrong with the. There's something called a tool config, right? The tool config is like the parameters and the understanding that the model has for the inputs and the actual functions and tools that it has. And then you have your functions, the Python functions themselves. They need to come back and structure JSON to be able to come back to the tool. All of this needs to be coded correctly in Troubleshot, right? So these are this is pretty much how you would troubleshoot it, and there's tools to do this. You can do this all through code as well.

Tim McConnaughy: 29:26

Perfect sense to me yeah.

Chris Miles: 29:28

I mean it sounds like you know potentially I guess it depends on your use case right and what your actual network implementation looks like but it sounds to me like there there's there's the need for a lot like a lot of work upfront, like there's a lot of coding that you need to do upfront to kind of get this stuff in place um before um before it's actually, you know, usable, depending on you know, like you said, if I'm assuming, if the network stack that you use supports an API that is relatively structured and has good documentation, you can probably just feed that into the agent. As far you know, and at least that's from what I got from our talk with John, you can kind of feed all that API documentation into the agent and it knows to use that tooling. But you know, when it comes to writing your own Python and things like that, how much do you think a network agent specifically requires kind of more upfront lifting compared to another activity that you use an agent for?

Du'An Lightfoot: 30:20

So networking compared to whether agent or? Yeah, exactly.

Chris Miles: 30:25

I mean like, yeah, I guess that's a very open-ended question. Very open-ended the use for an agent can be anything under the sun.

Tim McConnaughy: 30:31

I guess it can be anything under the sun.

Du'An Lightfoot: 30:33

it can be anything simple, yeah so I I think, um, when I like to look at this, when it comes to network automation, I always looked at it a little different than a lot of network engineers, and I think it was because of my time at a particular company we aggregate our data to a central location, and now this is before netbox, so now the netbox is around, they kind of aggregate you yeah, put it there right.

Tim McConnaughy: 30:54

We aggregate our data to a central location.

Du'An Lightfoot: 30:56

And now this is before NetBox. So now the NetBox is around, they kind of aggregate to there, right, and then you can access everything from there. So the first thing you probably want to do is aggregate your data, rather than going out and having your agents or LLMs making API calls to your devices, which that doesn't sound very safe right.

Du'An Lightfoot: 31:17

I mean, it can happen, but that's probably not a way we want to use it, right. Right, we want to make sure that it has read-only access right to a particular database or to a particular API, something where it can just read the data and then start there. That would be the simplest thing, right? And then seeing what you can accomplish with that right.

Chris Miles: 31:33

So I think you're kind of like really emphasizing the value of having a CMDB in some capacity about the network, rather than pulling the network directly, having that intermediary device or mechanism that's going to have all that state data so that you can kind of really simplify what the AI needs to reach out to in the first place.

Du'An Lightfoot: 31:53

Yeah, but we also have to think about, like the I would say, the scale of the network. That's something that's always got to come into mind, because if you've got 10 devices, how much value are you going to get out of all that? You know what I mean. So that's something we also have to think about. But if you're doing things at scale, yeah, aggregating your data and being able to centralize it, it's going to help and save you a tremendous amount of time and headache.

Chris Miles: 32:17

It'd be like that meme with the guy with the giant spoon and it's like my you know three three three-tier web app with cdn and it's like, it's like a tiny little ball, yeah yeah, I mean.

Tim McConnaughy: 32:29

So you actually, I mean you wrote an agent, right, like the network whisperer code is is an agent. So if anybody wants to see what Duan's talking about and actually get down in the nitty gritty of what he's talking about, like and we're going to have it in the show notes, of course, right, Um, the code's there, like straight up. Just go take a look at it and you'll between how much of the Python is there versus what we're involving the LLMs with and stuff right.

Du'An Lightfoot: 32:56

I think I'm going to redo it, because in order to run that code, you need access to AWS. So what I think I want to do and if I can do it, if it's still available, I'm going to rewrite it, maybe containerize it, put a front end on it. But the back end will probably connect to Cisco DevNet.

Du'An Lightfoot: 33:12

Okay, put a front end on it, but the back end will probably connect to like um cisco devnet right, grab the information from there and then everyone could have access to it and actually see an agent and be able to reverse engineer it and try it out for themselves. I think I'll do that in the next day or two and I'll get it to you and it'll be in the show notes oh, brilliant.

Tim McConnaughy: 33:27

Uh, I mean, that's what john did too, I mean, and we're we're big fans of the idea of containerizing this whole thing, because you know, especially for network engineers who don't already have all the hey, you got to install Python virtual environment, like that's the old DevNet, struggle right about getting people spun up. If you could just hand them a container and be like it's all preloaded, dude, just do the part that you care about. You know that's great, all right, just do the part that you care about. You know that's that's great, though. All right. Um, let's see, is it? No? I just, uh, I'm trying to think is there anything we? I mean there's so much, honestly, we could probably go god, freaking forever on this. This is so interesting stuff. I, I don't know. I find this very interesting and I I also I'm glad that you're here and I'm glad that we're friends and I can ask you all these questions, because it's hard, man, it's hard to even go to the llms and ask them these questions.

Du'An Lightfoot: 34:12

You know, like I think part of it is, um knowing the right, the questions to ask. Yeah, I've been kind of doing it and I understand, you know the thing, the struggles that you all have, you know in networking, and then also understand the llm side, since I do it so much. You know, um, but there's things that you all see today in networking that I don't see. So we got to kind of help each other. You know what I mean, yep.

Tim McConnaughy: 34:34

We all got to help each other. That is absolutely the case.

Chris Miles: 34:36

Yeah. I think maybe to close out, there was a great piece in your talk kind of talking about the road ahead and what the strategies are for adopting AI in this new era that we're in Right. So I mean specifically maybe you can kind of go over some of those points, what you would recommend to the listeners on where they should get started.

Du'An Lightfoot: 34:55

Yeah, I think network automation starting there with Python, is networking, of course, networking foundations and getting familiar with networking. I would say layer one, layer two, getting familiar with Python. Layer three will be APIs, because that's that networking layer, of course, right, getting familiar with that. I would say layer four would be kind of connecting it all together, start making code and doing some automation, and then that session layer is going to be that transition to understanding LLMs, right, being able to understand how to write prompts, understanding how RAG works, understanding the different types of models, right. And then from there, when we started looking at presentation and application, we're kind of building it all together. Maybe you're using something like Streamlit and you're starting your own type of, I would say, agent or agent to workflow by saying, okay, I have this Python code, I'm able to automate this. Now let me see if I could ask a chat bot to be able to run these tools and provide me a response. And then I think, if you kind of get that layered approach, then your wheels will get turning and you'll start building out your own solutions and being able to see where you can actually take this, because it's kind of hard to see it until you actually get your hands on it and try some things out, right. But if you don't really understand how to code, I recommend okay, use AI to help you learn how to code.

Du'An Lightfoot: 36:19

I use AI for so many things. It's the part when I'm driving home. I don't listen to music, I'm talking to AI trying to learn stock options or learn something you know because it can answer questions. And then I'm already studying these things and now, after I'm studying, I could talk to AI and ask it to clarify, and if it doesn't sound right, then I can say are you sure? And then it's going to say, yeah, I don't know You're absolutely right.

Tim McConnaughy: 36:43

I got this all wrong.

Du'An Lightfoot: 36:45

And then I know I need to do research. But then I find out what the limitations are and where the gaps are, and it helps me learn even more, because I'm realizing that it can't help me here and I got to dive deeper.

Tim McConnaughy: 36:56

Yeah, it's a big feedback loop. I guess that's a really good point, right. The more you learn from AI, but also just from interacting with it, you start to find the edges, like you know where, the edges of the coloring book that you can't go outside, or where it really is good at, and and that's a whole skill in and of itself, beyond just simply prompt engineering. So, yeah, it's a good, good call.

Chris Miles: 37:17

That's crazy. So you're just doing this. While you're driving, you're just talking to an agent yeah yeah, that's crazy man. Like what are you using to do that?

Du'An Lightfoot: 37:27

Um, yeah, I use the chat GB gbt app so I pay for probably about five ai tools. Right, but they are. They help me learn. Um yeah some of them are better than others, but I use one for code. I use multiple for code, right?

Tim McConnaughy: 37:44

yeah, um, and it's helped me grow oh, I did want to ask you one more thing before we, before we close out, and I it's important what do you think about specialized language, like small language models, like, do you think we're going to see more small language models, more focused small language models, kind of the bridge between getting all the way down to agents and, you know, all the way up to LLMs, to maybe save resources and get more specialized or not.

Du'An Lightfoot: 38:08

Yeah, I think when you start looking at these applications, a lot of organizations want to save on cost, and so, if you look at a smaller model, it's going to be lightweight, it's going to be fast. You got the 1B and 7B models from like Lama, which is meta. You got models from Alibaba that are pretty lightweight and small. You have hosted models on cloud with like haiku, which is pretty fast, and I think um openai just released their 03 mini. That's pretty fast and pretty smart, and so when we look at these models, um, a lot of people go after the larger models, right, the large language models, which a smaller model is a large language model as well, except it doesn't have as many parameters right right?

Du'An Lightfoot: 38:51

um, you can run that on your laptop, right, because it's lightweight. The capabilities on the large models have been distilled down to the smaller models, so they are optimized in this lightweight compact box Right To have some of the same performance as the larger models, but they may not have the depth and breadth of those larger models.

Tim McConnaughy: 39:12

Not as much information, yeah.

Du'An Lightfoot: 39:14

Right, but they understand language. So the thing that you could do to get around that provide context. So when we talk about a lot of the tasks that we want to perform, a lot of it is about the context that you wanted to understand your data and perform an action off the data. It's not necessarily what's inside the brain of the model, it's about the data that you have and if it has access to that data and you're structuring your products appropriately, it can actually get you some really good performance for a lot cheaper cost and be a lot faster in your application. Yeah, that's really important.

Tim McConnaughy: 39:47

I think in the future, go ahead, future, go ahead. No, I was just saying I think that's so important now because that's the cost of doing this is still out of the reach of a lot of organizations to like run this stuff locally, so I think that's the way forward. Sorry, I didn't mean to cut you off there.

Du'An Lightfoot: 40:00

No, I think if you're asking questions with the model out the box trying to see its capabilities, I don't think that's the right approach. I think more so right in a prompt that actually has the context that you want to provide it, like what do you want it to solve? Give it the information it needs to solve the problem and then evaluate the model right how does it perform, what is the consistency, what is the accuracy on providing the responses right. And then, once you determine that, it's like okay, this works for me, Because what I run now I'm running all smaller, smaller models, like everything that I want to do. I can do with a lightweight model, except for right code oh, is it just not?

Tim McConnaughy: 40:37

the? The skill, the, the reasoning of like how to create code just isn't in the small, or just the the depth of of code knowledge is just not there in the smaller code knowledge for the things that I do.

Du'An Lightfoot: 40:47

Um, gotcha, I go to like I use cloud sonnet 3.5 to. That's been the best model when it comes to code. But if I just need something to understand something really quick, there's a couple of applications I use for that, and on my local laptop I run something called Ollama which allows you to run large language model locally on your laptop, and so I run Ollama 3 on my laptop. I also run DeepSeek on my laptop. Well, I don't run DeepSeek on my laptop, I run that on my desktop because I have an NVIDIA card which is a 2060.

Chris Miles: 41:18

I was going to say be careful, they might go to jail. 2060. I was going to say be careful, they might go to jail.

Du'An Lightfoot: 41:23

Might go to jail Hawley.

Chris Miles: 41:27

He put together legislation trying to get people to go to jail for a maximum of 20 years. So no worries For running DeepSeek, they're trying to introduce this dude's.

Tim McConnaughy: 41:37

Just, he's one of those performative guys, but anyway, yeah, he just tried to introduce legislation. Yeah, if you run deep seek, if you download deep seek, it wasn't even run it like if you download deep seek, you can go to jail or something like that. I don't know that it's it's typical government, like how you guys even plan on enforcing this type of.

Du'An Lightfoot: 41:55

I wonder what that means, because there's like, again, there's deep, there's deep. Seek the steel models where they've been trained off of, like Lama Right Off of GPT, or trained off of QM. Yeah, you know. So it's like I don't know.

Tim McConnaughy: 42:09

It's typical. The government is trying its best, to you know, to be relevant in this stuff. But anyway they just don't know what they're talking about.

Chris Miles: 42:18

Last thing I'll say about the kind of expansion of these smaller models, and this is probably my just kind of dumb way of thinking about this, but the thing that worries about me, or worries me about this is if we start to see a vast implementation of these smaller models.

Chris Miles: 42:35

If I think about me as a person, when I'm going to make a decision, when I'm going to give a recommendation or process any data, I need a lot of context to make a decision, like I need to know the whole story, blah, blah, blah.

Chris Miles: 42:47

Even if I think about the concept of maybe, like, maybe you wanted a networking specific this is this is probably a dumb example, but if you, if you wanted a networking specific model that was just focused on computer networking and like it knew everything about networking in and out, blah, blah, blah, and you got it to make decisions without also understanding the context of how applications work and how applications actually function, like are we, are we, are we going to start snowballing and like, like, accelerate towards mediocrity in that, in that context, without having the full context of the situation? And you know what? Like I don't know. I like this is just something I'm spinning around in my head, I don't know if it makes any sense, but that's uh seems like that would worry me a bit I think if you needed something that specialized, maybe fine-tune the model yeah, I've heard of fine-tuning.

Tim McConnaughy: 43:32

I mean fine-tuning. I guess I was talking to andrew about that. I really uh, you know he's running the gni boot camp or whatnot. He was showing me how you can fine tune. It's interesting, interesting idea.

Du'An Lightfoot: 43:41

So fine tuning would be. You have this model, it's pretty capable, right, but it doesn't have domain specific information or intellectual property that's related to answering your questions and you really wanted to specialize in this, and a context window just isn't enough, ragging isn't getting it done and you don't have the money to train your own model from scratch. So what fine tuning does is it says okay, question, answer, response. You fine tune the model to be able to recognize the questions that you ask and how to respond when those questions are asked, right? So basically, you're taking all of your data and you're basically putting it in question and answer or question and response format and then you retrain the model based off of that information. So now, some of the knowledge that it didn't need is going to kick that out and add new knowledge based off of what you have. But it still understands language, it still has that capability, but it specializes in the information that you have.

Du'An Lightfoot: 44:38

Now, fine tuning is very specialized. It's not something that's OK. Let me take these questions, let me take these is very specialized. It's not something that's okay. Let me take these questions, let me take these responses, and then it's going to work.

Tim McConnaughy: 44:46

No, this is something that you have to iterate on and that to actually get it right yeah, there's, there's, so there's so much of there's so much that goes into the uh, the underpinning, uh mathematical stuff for the deep learning and machine learning, and all that really fools this but.

Du'An Lightfoot: 45:01

But just one more piece. Think of it this way right, if we do use the smaller models and they're not able to get you an accurate response? You can actually create a system that says, okay, if I don't know information, I need to go to my big brother and get the information right. You can actually create it, like all of this is through code.

Du'An Lightfoot: 45:18

Yeah, all of this is through code. Like you can save on cost by letting the smaller model handle the not as complex tasks and then defer to the larger models for those really more in-depth, complex tasks.

Tim McConnaughy: 45:31

That's a really good point. Man, that's a really good point. You know I've worked with Andrew Brown on the Gen AI boot camp stuff and we were working on the. We had a project we're working on which we'll see later on in the bootcamp, and that's exactly how it worked. Right, we tried to handle some of the stuff locally and then I didn't even know you could do it, but of course I should have thought of course it's just code right At the end of the day, right, you just make an API call, feed it a prompt and then get the response.

Du'An Lightfoot: 46:01

You know. It's something simple because you can tell a model. If you don't know, say I don't know, right, and if you don't know, call this function. Call Big Brother, right and you call. Big Brother and pass in the phone. Yeah, phone a friend, original prompt. Phone a friend Right and pass in the prompt to Big Brother.

Tim McConnaughy: 46:19

Big brother answers it and then helps little brother respond yeah, and you're only paying for the, the tokens, those, those very specialized tokens that you ran on on the big brother, right? So that's that's really good. Point too, man, that's really good, all right. Oh, man, we have a lot to think about. Uh, this has been so awesome. We're definitely gonna have to have you, uh, back again. I'm sorry in advance because you're gonna, you're gonna, but no man. Man, you're so good at explaining this. I mean, obviously that's your job, so no surprise there. But it's always great having you on man, and this has been so helpful and I hope that it's going to help a lot of our listeners understand their place kind of in the hierarchy to start playing with this stuff. So I appreciate it Anything to add?

Chris Miles: 47:01

Chris, Nothing to add. I will say one thing that I didn't know prior to meeting DeJuan at reInvent last year. Dude, you're tall as hell, bro. I didn't know you were that tall. Yeah, you always fool me. Every time I see you, you're sitting down, so I didn't know. I mean he was towering over me. You ever dunked a basketball? Can you dunk?

Du'An Lightfoot: 47:20

nah man, no, I could direct grab the rim when I was in shape.

Chris Miles: 47:26

Hey, touch a rim still something I've never done out of basic training.

Du'An Lightfoot: 47:29

I think I dunked like one time and then. I thought, I drank a beer and was down here from there awesome alright, we'll wrap up there again.

Tim McConnaughy: 47:40

Thank you to all the listeners for listening or for watching. If you're watching us on YouTube maybe both, maybe you want to watch this twice for some reason. If, hopefully, if you like what you heard, saw, then please subscribe to us on your podcatcher. Watch us on YouTube. Check out DeJuan's everything. We'll make sure all his stuff is in the show notes, but especially the Otacon stuff and the code that he's going to rewrite by the time you see it. So we'll see you on another episode. Thanks for joining us.

Chris Miles: 48:10

Hi everyone. It's Chris and this has been the Cables to Clouds podcast. Thanks for tuning in today. If you enjoyed our show, please subscribe to us in your favorite podcatcher, as well as subscribe and turn on notifications for our YouTube channel to be notified of all our new episodes. Follow us on socials at Cables to Clouds. You can also visit our website for all of the show notes at CablesToCloudscom. Thanks again for listening and see you next time.