Cary Millsap has been part of the Oracle community since 1989. He is the founder and president of Method R Corporation. He is widely known in the Oracle community as a speaker, educator, consultant, and writer. He wrote Optimizing Oracle Performance with Jeff Holt, and he is a co-author of Oracle Insights: Tales of the Oak Table. Cary is also an architect of the Method R software tools that help professionals around the world manage Oracle performance. He blogs at carymillsap.blogspot.com and tweets at twitter.com/CaryMillsap.
As a teacher, Cary has the rare ability to teach complex topics in a manner that everyone can understand. He is innovative and entertaining, and he always challenges his audiences to think beyond the status quo. Cary’s reputation as one of the world’s foremost experts in Oracle performance diagnosis is richly deserved.
Principal Software Engineer with Honors
How Can Cary Millsap Help You?
You can bring Cary Millsap’s passion and experience to your site, where he can help you in any of the following areas:
- Solving Oracle system performance problems.
- Implementing processes to prevent performance problems from occurring.
- Learning how the Oracle kernel performance instrumentation works, and how to use that instrumentation both for diagnosing and preventing performance problems.
- Learning the whys and hows of good software performance instrumentation that should occur during software development lifecycle.
- Implementing or learning how to use Method R performance management software.
- Designing specific Method R software enhancements to suit your needs.
Cary worked at Oracle Corporation from 1989 through 1999. In the early 1990s, he created Oracle’s OFA Standard, which became the default configuration standard for Oracle software installations worldwide. Many of the SQL scripts he distributed throughout his Oracle career (called “Oracle APS”) are still in use today. At Oracle, he accumulated various honors including Oracle Consulting’s “Consultant of the Year” award. He visited hundreds of clients on performance and architectural projects and spoke to thousands of architects, DBAs, and developers in courses, seminars, and user groups. He was a founding member of the Massive Open Systems Environment Standards group, where he worked directly with Oracle Development and Marketing to help shape features that Oracle would release in versions 7 through 11. When he departed Oracle in 1999, he was a Vice President in charge of 85 performance consultants and a 15-person service line group that constructed system architecture and system management services for Oracle Consulting practices worldwide. In 1999, he co-founded a company called Hotsos. In April 2008, Cary severed his partnership with Hotsos to begin a new company called Method R Corporation.
In the early 2000s, Cary pioneered the idea that a full and unambiguous account of an Oracle session’s end-user response time (called a profile) could be obtained from Oracle extended SQL trace data. The motive and means for obtaining this information helped to revolutionize the way Oracle performance analysts work. To explain the process, he wrote Optimizing Oracle Performance with Jeff Holt, which earned Oracle Magazine’s 2004 Author of the Year award. He led the design and development of the Hotsos Profiler software package, which automates the most difficult part of implementing Method R for Oracle applications. He created and taught courses and seminars to Oracle practitioners at events all over the world. He co-founded the Oak Table Network, a loose association of like-minded people who make their livings from their expertise with the Oracle database engine and their skill at handling performance issues. In 2007, Oracle Corporation granted him the status of Oracle ACE Director. In 2010, he was honored with the Oracle Development Tools User Group “Editor’s Choice” 2010 award for his paper, “Thinking Clearly about Performance.” This paper was also featured in both ACM Queue and Communications of the ACM magazines.
During production of Optimizing Oracle Performance, Cary established and proved the idea that the location of the “knee” in an M/M/m queueing system’s response time curve depends only upon the system’s number of parallel service channels and is in fact independent of the arrival rate and service rate. This result makes it possible to plan computer capacity requirements with much simpler models than practitioners had tried to use in the past.
Cary Millsap Presentations
Here is a sampling of some of the presentation topics that Cary Millsap presents at conferences, user groups, and customer sites all over the world. Topics range in duration from 1 to 4 hours. To book Cary Millsap for a speaking engagement, please mail us directly at .
Learning about Life through Business and Software (non-technical keynote)
I run a software business, I help my wife raise three children, and I like to think. I especially like to think in metaphors—to learn in one area and then apply that knowledge in another area that seems completely unlrelated. Raising children and running a business put everything in my mind to use; in fact, the jobs push me beyond my own capacity sometimes. In this talk, I’ll share some of the ideas that I’ve learned about raising a family from running my business, and vice versa. I’ll talk about why thinking clearly is more important than the right answer, how testing in schools should be more like good software testing, and what I’ve learned about the pursuit of happiness.
Thinking Clearly About Performance
Creating high-performance as an attribute of complex software is extremely difficult business for developers, technology administrators, architects, system analysts, and project managers. However, by understanding some fundamental principles, performance problem solving and prevention can be made far simpler and more reliable. This paper describes those principles, linking them together in a coherent journey covering the goals, the terms, the tools, and the decisions that you need to maximize your application’s chance of having a long, productive, high-performance life. Examples in this paper touch upon Oracle experiences, but the scope of the paper is not restricted to Oracle products.
This paper was honored with the Oracle Development Tools User Group “Editor’s Choice” 2010 award and is featured in ACM Queue and Communications of the ACM magazines.
Open Discussion about Oracle Performance
With over 22 years of Oracle experience—and conversations with thousands of Oracle practitioners throughout the world—to draw from, Cary Millsap hosts open discussion with small or large groups to answer questions and explore aspects of Oracle performance that you may never have considered.
Performance is a Feature: Here is the Specification
To many software developers, designers, and architects “performance” is a side-effect, ...an afterthought of designing and building proper features like “book an order” or “look up a book by author.” But great performance at scale doesn’t happen by accident. The first step is to know what performance is: it is the answer to the question, “What have people been experiencing?” Knowing what people experience when they use your software is possible only if you treat performance as a proper feature, a feature you analyze, design, build, test, and maintain. This session explains the steps that will get you started.
Instrumentation: Why You Should Bother
In our increasingly virtualized environments, it is ever more difficult to diagnose application defects—especially performance defects that affect response time or throughput expectations. Runtime diagnosis of defects can be an unbearably complicated problem to solve once the application is sealed up and put into production use. But having excellent runtime diagnostics is surprisingly easy if you design the diagnostic features into the application from its inception, as it is being grown, like you would with any other desired application feature.
Oracle’s Extended SQL Trace Data for Developers
Oracle’s extended SQL trace data stream contains a linear sequential record of every database call and every operating system call that the Oracle kernel executes in response to the code that you write. The trace file contains timings, which enable you to precisely measure how long your code takes to run and why. The detailed information about individual dbcalls and syscalls is vital to building scalable applications (as well as troubleshooting them in production). The information in there is a gold mine, and some of it is available nowhere else.
For Developers: Making Friends with the Oracle Database
To many application developers, an Oracle database is just a “data store” with an API that they call when they need to persist an object. It’s a helpful abstraction for managing functional complexity, but it can lead to some horrible performance problems. You can avoid those problems by better understanding what’s going on inside the Oracle kernel. It’s not that hard to do. The key is understanding how to measure how your code spends time inside Oracle. Once you’ve done that, your response time profile leads you exactly to your performance improvement opportunities.
My Case for Agile Methods
Among many of my Oracle database administrator (DBA) friends, “agile” is widely regarded as a dirty word, a synonym for “sloppy.” However, adopting the principles of the Agile Manifesto (specifically, the implementation of the Agile Manifesto called Extreme Programming, or XP) has radically improved the commercial and technical success of projects that I’ve worked on. Agile principles have enriched my entire life—not just professionally, but personally. The contradiction between the typical DBA’s perception of “agile” and my own is profound. This paper describes my experiences with Agile values and my implementation of them. I describe the circumstances that have led me to believe passionately that it’s XP that will best assure the success of my projects. I describe what has worked for me and why, and I describe what hasn’t worked and why.
Millsap’s Grand Unified Theory of “Tuning”
First there were the v$ and x$ views. And tkprof. Then came bstat/estat, Statspack, ASH, ADDM, OEM, and plenty of third-party tools that look kind of like those tools. There are loads of OS tools too that have been around a long time, like top and sar and strace and pstack and gprof. And then there are the methods: OPIM, and YAPP, and Method R to name three. In this presentation, Cary Millsap gives a brief tour of the moving parts and reveals his own perspective about how all these parts should fit together.
Improving Developer and Analyst Productivity with Method R Software Tools
As the lead designer and developer (as well as a devoted user) of Method R software tools, Cary Millsap is uniquely positioned to describe how to use those tools to improve developer and analyst productivity in the quest for response time and throughput optimization. In this session, Cary introduces audiences to zero-click tracing with the Method R Trace extension for Oracle SQL Developer, response time profiling with the Method R Profiler, and managing trace files with Method R Tools.
Messed-Up Apps: a Study of Performance Antipatterns
Sometimes, performance is not a matter of how you write your code; sometimes, your performance is doomed from the very design specification. In this session, Cary Millsap presents examples of applications that perform poorly and the patterns that result in reliably poor performance, no matter what development language you’re using.
Optimizing for Oracle: When? How?
If premature performance optimization is a root of evil—and Cary Millsap believes it is—then developers face a difficult question: When and how are you supposed to optimize? If you optimize prematurely, you face the consequences of investing resources that are easy to prove after the fact to have been wasted. If you optimize too late, you create serious jeopardy for your customers and yourself as your applications fail to scale up to the user loads they are required to support. In this session, Cary offers a lifeline: a process and a set of guidelines that will allow you to reduce your risk of post-production performance crises, while not requiring you to overinvest into performance apprehensions that may never materialize.
Oracle Operational Timing Data
The most important statistics for the Oracle performance analyst are the Oracle kernel’s timing statistics. Yet, an amazing number of misunderstandings about Oracle timing statistics prevail in the market today. As a result, analysts can fail to see potential performance improvements that are right under their noses! This paper explains how the Oracle kernel measures and reports its own timings. It explores some of the ironies of digital clocks that might have diminished your confidence in Oracle’s timings. It explains how to get past these ironies to prove more assertions about Oracle performance than you can today. Finally, it discusses why the V$ data that you probably trust today can easily lead you down a rat-hole. The result is an ability to improve system performance beyond the levels at which you might be stuck today.
What the Oracle Timed Events Mean
Many Oracle Database administrators and application developers are familiar with what Oracle calls its “wait interface.” What people tend not to understand so well is that each timed event reported by the “Oracle wait interface” actually maps to one or more operating system calls (syscalls). By consulting your operating system documentation for these syscalls, you can earn a better understanding of what the Oracle Database does than if you just read your Oracle documentation. In this presentation, Cary Millsap shows how to use commands like truss to learn which syscall(s) each Oracle timed event maps to, and he discusses the meanings of the most common events you’ll encounter.
How to Make Application Performance Easy to Diagnose
Diagnostic data collection can be the most difficult step in any system performance improvement project. But the collection of good diagnostic data doesn’t need to be difficult. The determining feature is the application itself. The key to making application performance easy to diagnose lies in the hands of the application developer. This session shows how a few simple development decisions can make application performance easy to diagnose.
Why You Can’t See Your Real Performance Problems
Reflecting across nearly 20 years of solving Oracle performance problems, Cary Millsap has recognized a single pattern of behavior that is the dominant reason for failure in all the projects he has witnessed. In almost every case he has seen, failures in diagnosing and repairing performance problems have been caused by unrecognized skew in diagnostic data. This presentation shows several examples that illustrate why skew is such a pervasive problem for performance analysts.
Repairing Oracle Performance Problems
In 2003, Cary Millsap and Jeff Holt published a new method for diagnosing and repairing Oracle system performance problems (Optimizing Oracle Performance: O’Reilly). Users of what the book calls Method R (the ‘R’ stands for “Response time”) routinely fix performance problems that had evaded repair (and sometimes even detection!) by other methods for months or more. This session demonstrates how Method R works and how people use it to diagnose and repair system performance problems in perfect alignment with business priorities. The case study uses Oracle diagnostic data specifically to show how even very difficult performance problems relent quickly and permanently to the method.
Preventing Oracle Performance Problems
In 2003, Cary Millsap and Jeff Holt published a new method for diagnosing and repairing Oracle system performance problems (Optimizing Oracle Performance: O’Reilly). After solving the important diagnostic problem, their research turned to the subject of preventing such problems from occurring in the first place. The ultimate goal is a method through which you can always understand what your next performance problem will be so that you can diagnose and repair the problem before it ever actualizes as an end-user experience. This session summarizes the progress of their research so far.
Measure Once, Cut Twice
“Measure Twice, Cut Once” is a reminder that careful planning yields better gratification than going too quickly into operations that can’t be undone. Sometimes, however, it’s better to Measure Once, Cut Twice. It’s one of the secrets behind how carpenters hang square cabinets in not-so-square kitchens. And it’s one of the secrets behind how developers write applications that are easy to fix when they cause performance problems in production use. The key is to know which details you can plan for directly, and which details you simply can’t know and therefore have to defend yourself against. In this session, Cary will discuss some aspects of software development where flexible design is more important than detailed planning, using woodworking analogies for inspiration. He will describe some particular flexibilities that your software needs, and he’ll describe how to create them.
Accountability for System Performance (introducing Six Sigma quality in software performance management)
Businesses are commonly organized in such a manner that the departments held accountable for making application systems fast and efficient don’t have enough leverage to get the job done. The result is painful. Users suffer from slow applications, and system costs spiral while organizations expend energy deflecting blame. This session describes how the right measurements and a few process changes can produce a culture of performance accountability across departments, resulting in faster, cheaper, more efficient systems.