Cary Millsap has been part of the Oracle community since 1989. He is the founder and president of Method R Corporation and a performance specialist at Accenture Enkitec Group. He is widely known in the Oracle community as a speaker, educator, consultant, and writer. He wrote Optimizing Oracle Performance with Jeff Holt, and he is a co-author of Oracle Insights: Tales of the Oak Table. Cary is also an architect of the Method R software tools that help professionals around the world manage Oracle performance. He blogs at carymillsap.blogspot.com and tweets at twitter.com/CaryMillsap.
As a teacher, Cary has the rare ability to teach complex topics in a manner that everyone can understand. He is innovative and entertaining, and he always challenges his audiences to think beyond the status quo. Cary’s reputation as one of the world’s foremost experts in Oracle performance diagnosis is richly deserved.
Principal Software Engineer with Honors
How Can Cary Millsap Help You?
You can bring Cary Millsap’s passion and experience to your site, where he can help you in any of the following areas:
- Solving Oracle system performance problems.
- Implementing processes to prevent performance problems from occurring.
- Learning how the Oracle kernel performance instrumentation works, and how to use that instrumentation both for diagnosing and preventing performance problems.
- Learning the whys and hows of good software performance instrumentation that should occur during software development lifecycle.
- Implementing or learning how to use Method R performance management software.
- Designing specific Method R software enhancements to suit your needs.
Cary worked at Oracle Corporation from 1989 through 1999. In the early 1990s, he created Oracle’s OFA Standard, which became the default configuration standard for Oracle software installations worldwide. Many of the SQL scripts he distributed throughout his Oracle career (called “Oracle APS”) are still in use today. At Oracle, he accumulated various honors including Oracle Consulting’s “Consultant of the Year” award. He visited hundreds of clients on performance and architectural projects and spoke to thousands of architects, DBAs, and developers in courses, seminars, and user groups. He was a founding member of the Massive Open Systems Environment Standards group, where he worked directly with Oracle Development and Marketing to help shape features that Oracle would release in versions 7 through 11. When he departed Oracle in 1999, he was a Vice President in charge of 85 performance consultants and a 15-person service line group that constructed system architecture and system management services for Oracle Consulting practices worldwide. In 1999, he co-founded a company called Hotsos. In April 2008, Cary severed his partnership with Hotsos to begin a new company called Method R Corporation. Today, Cary is runs Method R and works as a performance specialist for Kerry Osborne’s Accenture Enkitec Group.
In the early 2000s, Cary pioneered the idea that a full and unambiguous account of an Oracle session’s end-user response time (called a profile) could be obtained from Oracle extended SQL trace data. The motive and means for obtaining this information helped to revolutionize the way Oracle performance analysts work. To explain the process, he wrote Optimizing Oracle Performance with Jeff Holt, which earned Oracle Magazine’s 2004 Author of the Year award. He led the design and development of the Hotsos Profiler software package, which automates the most difficult part of implementing Method R for Oracle applications. He created and taught courses and seminars to Oracle practitioners at events all over the world. He co-founded the Oak Table Network, a loose association of like-minded people who make their livings from their expertise with the Oracle database engine and their skill at handling performance issues. In 2007, Oracle Corporation granted him the status of Oracle ACE Director. In 2010, he was honored with the Oracle Development Tools User Group “Editor’s Choice” 2010 award for his paper, “Thinking Clearly about Performance.” This paper was also featured in both ACM Queue and Communications of the ACM magazines. In 2011, he published the book The Method R Guide to Mastering Oracle Trace Data.
During production of Optimizing Oracle Performance, Cary established and proved the idea that the location of the “knee” in an M/M/m queueing system’s response time curve depends only upon the system’s number of parallel service channels and is in fact independent of the arrival rate and service rate. This result makes it possible to plan computer capacity requirements with much simpler models than practitioners had tried to use in the past.
Cary Millsap Presentations
Here is a sampling of some of the presentation topics that Cary Millsap presents at conferences, user groups, and customer sites all over the world. Topics range in duration from 1 to 4 hours. To book Cary Millsap for a speaking engagement, please mail us directly at .
The Most Important Things You Need to Know about Oracle
Cary Millsap has been a performance specialist and business leader in the Oracle ecosystem since 1989. In the years since, he has helped convert a lot of catastrophes into successes, and he has traveled with most of the best Oracle experts in the world. In this presentation intended for both technical and non-technical audience members, Cary will distill the most important lessons he has learned and illustrate those lessons with stories from his career.
Oceans, Islands, and Rivers: the mistake everyone always makes with trace files
When you execute a business task on a computer system, you create an experience. ...An experience between you and the machine. The duration of this experience is called response time. A sequence diagram helps you understand how that response time was consumed. A profile is a useful aggregation of the sequence diagram. A performance analyst looks at your time consumptions to determine whether it is possible to reduce the response time of the experience. ...And, if so, then by how much. The richest and easiest diagnostic information to obtain in this whole technology stack is available from the Oracle Database tier. ...Oracle’s extended SQL trace data. But in almost 100% of first tries with using Oracle extended SQL trace data, people make a data collection mistake that complicates their analysis. This is the story of that mistake.
Selected Cases in Software Performance
In this session, Cary Millsap speaks about performance topics from his experiences, on topics like these:
- Just STOP. The fastest way to do anything is to find a way not to do it at all. It’s a fun way of expressing Rule One of system optimization: filter early. This segment provides an interesting example of how you an application developer can filter even earlier than you might have thought, by redesigning a feature just ever so slightly.
- “Bind Variables” are not the Solution. Everybody knows the famous advice about using bind variables. But, as it turns out, it’s not really the right advice. Well, maybe "you should use bind variables" is the truth, but it’s not the whole truth. This segment explains why.
- How an Expensive Hardware Upgrade Can Make Your Application Run Slower. It’s the weirdest thing: you’ve spent the whole weekend (and bales of cash) upgrading your slow CPUs to faster ones, and now your system is slower than it was before. It happens more often than you’d think. Nobody expects it, but it’s actually easy to understand how it happens, and you can know beforehand whether you’re vulnerable.
My Whole System is Slow. Now What?
Your system is slow, and your whole business is suffering. If you can’t find it and fix it by Friday, then there’s not going to be a Monday. This horrific situation for a business is a tremendous opportunity for a performance specialist. So, where to begin? Step One in a situation like this is to discover what that problem is that’s causing so much pain, right? Wrong. It’s tempting to believe that there’s a single problem lurking inside your system that’s slowing everybody down, but the real world almost never works like that. Your most important first step is probably counterintuitive unless you’ve fixed systems like this a lot of times before. In this session, I’ll explain what this first step is and why it’s so vital that you get it right.
Learning about Life through Business and Software, a non-technical keynote
I run a software business, I help my wife raise three children, and I like to think. I especially like to think in metaphors—to learn in one area and then apply that knowledge in another area that seems completely unrelated. Raising children and running a business put everything in my mind to use; in fact, the jobs push me beyond my own capacity sometimes. In this talk, I’ll share some of the ideas that I’ve learned about raising a family from running my business, and vice versa. I’ll talk about why thinking clearly is more important than the right answer, how testing in schools should be more like good software testing, and what I’ve learned about the pursuit of happiness.
Thinking Clearly About Performance
Creating high-performance as an attribute of complex software is extremely difficult business for developers, technology administrators, architects, system analysts, and project managers. However, by understanding some fundamental principles, performance problem solving and prevention can be made far simpler and more reliable. This paper describes those principles, linking them together in a coherent journey covering the goals, the terms, the tools, and the decisions that you need to maximize your application’s chance of having a long, productive, high-performance life. Examples in this paper touch upon Oracle experiences, but the scope of the paper is not restricted to Oracle products.
This paper was honored with the Oracle Development Tools User Group “Editor’s Choice” 2010 award and is featured in ACM Queue and Communications of the ACM magazines.
Open Discussion about Oracle Performance
With over 22 years of Oracle experience—and conversations with thousands of Oracle practitioners throughout the world—to draw from, Cary Millsap hosts open discussion with small or large groups to answer questions and explore aspects of Oracle performance that you may never have considered.
Performance is a Feature: Here is the Specification
To many software developers, designers, and architects “performance” is a side-effect, ...an afterthought of designing and building proper features like “book an order” or “look up a book by author.” But great performance at scale doesn’t happen by accident. The first step is to know what performance is: it is the answer to the question, “What have people been experiencing?” Knowing what people experience when they use your software is possible only if you treat performance as a proper feature, a feature you analyze, design, build, test, and maintain. This session explains the steps that will get you started.
Instrumentation: Why You Should Bother
In our increasingly virtualized environments, it is ever more difficult to diagnose application defects—especially performance defects that affect response time or throughput expectations. Runtime diagnosis of defects can be an unbearably complicated problem to solve once the application is sealed up and put into production use. But having excellent runtime diagnostics is surprisingly easy if you design the diagnostic features into the application from its inception, as it is being grown, like you would with any other desired application feature.
Oracle’s Extended SQL Trace Data for Developers
Oracle’s extended SQL trace data stream contains a linear sequential record of every database call and every operating system call that the Oracle kernel executes in response to the code that you write. The trace file contains timings, which enable you to precisely measure how long your code takes to run and why. The detailed information about individual dbcalls and syscalls is vital to building scalable applications (as well as troubleshooting them in production). The information in there is a gold mine, and some of it is available nowhere else.
For Developers: Making Friends with the Oracle Database
To many application developers, an Oracle database is just a “data store” with an API that they call when they need to persist an object. It’s a helpful abstraction for managing functional complexity, but it can lead to some horrible performance problems. You can avoid those problems by better understanding what’s going on inside the Oracle kernel. It’s not that hard to do. The key is understanding how to measure how your code spends time inside Oracle. Once you’ve done that, your response time profile leads you exactly to your performance improvement opportunities.
My Case for Agile Methods
Among many of my Oracle database administrator (DBA) friends, “agile” is widely regarded as a dirty word, a synonym for “sloppy.” However, adopting the principles of the Agile Manifesto (specifically, the implementation of the Agile Manifesto called Extreme Programming, or XP) has radically improved the commercial and technical success of projects that I’ve worked on. Agile principles have enriched my entire life—not just professionally, but personally. The contradiction between the typical DBA’s perception of “agile” and my own is profound. This paper describes my experiences with Agile values and my implementation of them. I describe the circumstances that have led me to believe passionately that it’s XP that will best assure the success of my projects. I describe what has worked for me and why, and I describe what hasn’t worked and why.
Millsap’s Grand Unified Theory of “Tuning”
First there were the v$ and x$ views. And tkprof. Then came bstat/estat, Statspack, ASH, ADDM, OEM, and plenty of third-party tools that look kind of like those tools. There are loads of OS tools too that have been around a long time, like top and sar and strace and pstack and gprof. And then there are the methods: OPIM, and YAPP, and Method R to name three. In this presentation, Cary Millsap gives a brief tour of the moving parts and reveals his own perspective about how all these parts should fit together.
Improving Developer and Analyst Productivity with Method R Software Tools
As the lead designer and developer (as well as a devoted user) of Method R software tools, Cary Millsap is uniquely positioned to describe how to use those tools to improve developer and analyst productivity in the quest for response time and throughput optimization. In this session, Cary introduces audiences to zero-click tracing with the Method R Trace extension for Oracle SQL Developer, response time profiling with the Method R Profiler, and managing trace files with Method R Tools.
Messed-Up Apps: a Study of Performance Antipatterns
Sometimes, performance is not a matter of how you write your code; sometimes, your performance is doomed from the very design specification. In this session, Cary Millsap presents examples of applications that perform poorly and the patterns that result in reliably poor performance, no matter what development language you’re using.
Optimizing for Oracle: When? How?
If premature performance optimization is a root of evil—and Cary Millsap believes it is—then developers face a difficult question: When and how are you supposed to optimize? If you optimize prematurely, you face the consequences of investing resources that are easy to prove after the fact to have been wasted. If you optimize too late, you create serious jeopardy for your customers and yourself as your applications fail to scale up to the user loads they are required to support. In this session, Cary offers a lifeline: a process and a set of guidelines that will allow you to reduce your risk of post-production performance crises, while not requiring you to overinvest into performance apprehensions that may never materialize.
Oracle Operational Timing Data
The most important statistics for the Oracle performance analyst are the Oracle kernel’s timing statistics. Yet, an amazing number of misunderstandings about Oracle timing statistics prevail in the market today. As a result, analysts can fail to see potential performance improvements that are right under their noses! This paper explains how the Oracle kernel measures and reports its own timings. It explores some of the ironies of digital clocks that might have diminished your confidence in Oracle’s timings. It explains how to get past these ironies to prove more assertions about Oracle performance than you can today. Finally, it discusses why the V$ data that you probably trust today can easily lead you down a rat-hole. The result is an ability to improve system performance beyond the levels at which you might be stuck today.
What the Oracle Timed Events Mean
Many Oracle Database administrators and application developers are familiar with what Oracle calls its “wait interface.” What people tend not to understand so well is that each timed event reported by the “Oracle wait interface” actually maps to one or more operating system calls (syscalls). By consulting your operating system documentation for these syscalls, you can earn a better understanding of what the Oracle Database does than if you just read your Oracle documentation. In this presentation, Cary Millsap shows how to use commands like truss to learn which syscall(s) each Oracle timed event maps to, and he discusses the meanings of the most common events you’ll encounter.
How to Make Application Performance Easy to Diagnose
Diagnostic data collection can be the most difficult step in any system performance improvement project. But the collection of good diagnostic data doesn’t need to be difficult. The determining feature is the application itself. The key to making application performance easy to diagnose lies in the hands of the application developer. This session shows how a few simple development decisions can make application performance easy to diagnose.
Why You Can’t See Your Real Performance Problems
Reflecting across nearly 20 years of solving Oracle performance problems, Cary Millsap has recognized a single pattern of behavior that is the dominant reason for failure in all the projects he has witnessed. In almost every case he has seen, failures in diagnosing and repairing performance problems have been caused by unrecognized skew in diagnostic data. This presentation shows several examples that illustrate why skew is such a pervasive problem for performance analysts.
Repairing Oracle Performance Problems
In 2003, Cary Millsap and Jeff Holt published a new method for diagnosing and repairing Oracle system performance problems (Optimizing Oracle Performance: O’Reilly). Users of what the book calls Method R (the ‘R’ stands for “Response time”) routinely fix performance problems that had evaded repair (and sometimes even detection!) by other methods for months or more. This session demonstrates how Method R works and how people use it to diagnose and repair system performance problems in perfect alignment with business priorities. The case study uses Oracle diagnostic data specifically to show how even very difficult performance problems relent quickly and permanently to the method.
Preventing Oracle Performance Problems
In 2003, Cary Millsap and Jeff Holt published a new method for diagnosing and repairing Oracle system performance problems (Optimizing Oracle Performance: O’Reilly). After solving the important diagnostic problem, their research turned to the subject of preventing such problems from occurring in the first place. The ultimate goal is a method through which you can always understand what your next performance problem will be so that you can diagnose and repair the problem before it ever actualizes as an end-user experience. This session summarizes the progress of their research so far.
Measure Once, Cut Twice
“Measure Twice, Cut Once” is a reminder that careful planning yields better gratification than going too quickly into operations that can’t be undone. Sometimes, however, it’s better to Measure Once, Cut Twice. It’s one of the secrets behind how carpenters hang square cabinets in not-so-square kitchens. And it’s one of the secrets behind how developers write applications that are easy to fix when they cause performance problems in production use. The key is to know which details you can plan for directly, and which details you simply can’t know and therefore have to defend yourself against. In this session, Cary will discuss some aspects of software development where flexible design is more important than detailed planning, using woodworking analogies for inspiration. He will describe some particular flexibilities that your software needs, and he’ll describe how to create them.
Accountability for System Performance (introducing Six Sigma quality in software performance management)
Businesses are commonly organized in such a manner that the departments held accountable for making application systems fast and efficient don’t have enough leverage to get the job done. The result is painful. Users suffer from slow applications, and system costs spiral while organizations expend energy deflecting blame. This session describes how the right measurements and a few process changes can produce a culture of performance accountability across departments, resulting in faster, cheaper, more efficient systems.