Adapting technological tools to businesses needs
Adapting Technological Tools To Businesses Needs
It was a pleasure for me to work with all the wonderful people in our office here in San Francisco. First of all, I would like to thank Yannick Germain for being a great advisor. His ideas and tremendous support had a major influence on this thesis. He spent a lot of time helping me. I learned a lot during this time and I am convinced that this knowledge will help me in the future.
I would like to thank Natasha Douglas for reviewing my thesis. I am happy to have such a supportive friend. I enjoyed her interest in my research as well as the fruitful discussions.
My thanks to my friends and colleagues for the great time I had in our group. I enjoyed the atmosphere, their friendship and their support. My thanks to Laurent Buhler, Loic Gervais and Steven Lowenthal for the great collaboration over the months. It was a pleasure to work with all these people and to benefit from their knowledge. Especially, I would like to thank Laurent Buhler again for the interesting and fruitful discussions. My thanks to Marco Poggio, Linda Ferrari, and Daniel Meier for their collaboration while writing their diploma theses in our group.
My thanks to Frederic Schlosser for proof-reading several papers and for providing helpful suggestions for improving this manuscript. Any "linguistic crimes" are mine alone. My thanks also to Youssef Chachia for proof-reading parts of this document. My thanks to Sébastien Talukder for being open-minded to questions.My thanks to Jean Kruger for providing me his code on trees filtering.
My thanks to Vaolele Stawiarski, Lisa Tenorio and Julia Sprague for help on administrative matters and Google for the technical support.
Last but not least, I wish to thank my family who has always supported me, Lucie and Lena as well as Vincent and Christophe for real friendship.
With the current economical situation, no company would ever grow without a Client Relationship Management (CRM) software. Siebel System developed the first version of the Siebel CRM in 1993. Siebel System was bought by Oracle in 2005.
By the late 90's Siebel System was the dominant CRM vendor, peaking at 45% market share in 2002. It now has lost a large amount of this share, but it' and SAP are still listed as the top CRM vendors.
But Siebel's CRM (recently renamed Oracle CRM On Demand) still requires a lot of tuning. As a matter of fact, just like any big application, Siebel's CRM has downtimes. And when it's not entirely down, it's often slow or simply not fast enough. For this very reason, Germain CG developed a monitoring tool for Siebel. Its final purpose is to adapt a technological tool (Siebel's CRM) to the needs of businesses by improving its overall performance, eliminating application downtimes, pro-actively rooting out the causes of the issues, therefore improving the end user adoption. Germain CG's tools are helping businesses making the most of money they invested in Siebel's CRM.
During my final year of school at Supinfo, I have had the chance to be a part of this adventure. I have worked for 6 months for Germain CG and have learned a lot from this experience.
In this document, I am going to present you the issues that have to be dealt with when creating such a company and how to reach the announced goal of improving the businesses' efficiency by improving Siebel's CRM performance. I will then explain the context of these issues. Then, I'll reveal why I chose this subject and how the issues are usually solved mainly by exposing our competitors' products. Then I will layout the details of how I solved the issues. Finally, I'll explain why I chose this approach. I'll also try to be critical about this present document and explain you where I stand now.
3. Organizational chart
4. The Office
The office is located in the heart of San Francisco, right next to the famous Transamerica pyramidal tower in the financial district.
5. Company history
The business started 3 years ago when Yannick Germain decided to quit Siebel when it was bought by Oracle.
Yannick Germain is a professional with over 10 years of experience in software development, system performance and scalability engineering. This experience includes 8 years at Siebel Systems.
The main purpose of Germain CG is to provide monitoring tools and assistance to companies that are using the Siebel's CRM. Essentially, we help Siebel's customers improve the Siebel performance and avoid downtimes, which are very expensive.
Germain CG was founded in 2006. 12 people have been involved in building up Germain CG. Our staff ranges from board advisory members to software engineers, architect and product managers. Germain CG has been 4 years in business. Our 4 customers have all become Germain CG's references (at CIO level). They are located worldwide.
II. Enunciation of the issues
1. Technical issues
a) Usage of Siebel in companies
Siebel has so many features that, quite often, companies are not using them all. For that reason, it can be complicated for us, as a Siebel monitoring company, to make sure we meet our clients' needs. And to make things more complicated, Siebel offers a customization tool, which allows its clients to add features, improve the general performance or simply customize Siebel to the customer's needs.
b) Siebel's inherent problems
Siebel is a complete software suite, and since it's very difficult to customize it completely, it often generates a lot of errors. There are many types of errors with different severity levels. Our goal is to make sure our client's support team notices most of those errors, so that they can fix issues before they affect the client's business.
As a matter of fact, if Siebel or one of its components is down, the company can lose a lot of money.
From our developmental point of view, Siebel's errors can become very painful. Since Siebel is not open source, we have no way of predicting its behavior. That leads us to a very empirical way of developing. This is very dangerous because our reliability is at stake. For this reason, most of Germain CG tools are Siebel version dependant. That means we need to write a new version of the software every time a new version of Siebel comes out. Of course, some of our features are generic and can be adapted to any version of Siebel.
2. Business issues
a) Getting an income
Getting an income as a software startup company can be very difficult, and the financial crisis didn't help. Germain CG is lucky enough to be self financed. To start earning money, the company needed to sign business contracts with clients. That requires finding the clients. As a matter of fact, it's not that simple to find them. And it's even more complicated to sign the contract.
First of all, Germain CG's potential customers are only Siebel customers. That means we are entirely dependent on Siebel's sales.
Then, because of our main monitoring purpose, we are only marketable to companies that have Siebel and are not satisfied with its performance. There are lots of companies that fall into this category because for most of them, Siebel's support service is not good enough and doesn't provide a real monitoring tool like we do.
Finally, to actually sign the contract, we need to go through a certain amount of steps.
- Meet the right people,
- Show them a live demo of our software.
- Create the proof of concept on their own environment.
Each of those steps can fail. And because it's so difficult to start from scratch with a potential customer, another main business issue is to get enough recommendations from Siebel and our current clients.
b) Getting recommendations from Oracle & clients
Because many potential clients would simply answer "it looks interesting, but we need proof that it really works on a bigger scale", one of our top priorities was to be able to cosign a whitepaper with our only current customer. The Netapp & Germain CG success story is a document that proves how valuable Germain CG tools have been to Netapp's Siebel implementation. See annex.
Another great way of selling our software is to have Oracle (Siebel's current main shareholder) help us sell the software directly, when their salesmen are selling the Siebel CRM. Being recommended by Oracle would make sure that our potential clients find our software reliable enough and convince them that it adds a great value to Siebel.
a) Technical challenges
- Learning how Siebel works
- Learning how Germain tools work
Learning how Siebel works is such a long process that I have the feeling I'll never be done with it. However, Siebel's CRM, on its user interface level, is quite easy to understand. It becomes extremely complicated when dealing with log files, SARM files or in general with anything that is related to the multi server architecture.
Germain tools, even if they're not anywhere near as complicated as Siebel's CRM were, made up of quite a number of programs before I started working at Germain CG (and still are). And each of them is linked to a bunch of files, not often well documented. This can be explained by the fact that before my arrival, only one full time developer was working on these tools.
b) Business challenges
Ø Finding clients
- Siebel contacts
- Word of mouth in Siebel contacts
- Oracle Open World '09
In Germain CG's case, the Yannick Germain's contacts at Siebel System were great help. Since a lot of his former coworkers have left Siebel Systems at the same time that he did, a lot of them ended up working for companies that are using the Siebel CRM. Therefore, the connection to these companies was much easier. He only had to keep in touch with his former coworkers to know how the new company they were working for was doing and if they would be interested in buying a monitoring tool for Siebel.
The former coworkers that are now working for Oracle are also a huge value for Germain CG's promotion. As a matter of fact, these people are in the hearth of the action and can eventually recognize Germain CG's efficiency officially. That is, as explained above in section "Getting recommendations from Oracle & clients", a tremendous value for Germain CG.
Germain CG's website has also a big value regarding the number of prospects the company can have. It shows up in the 5 first results when looking up "Siebel monitoring". But for security reasons, the website itself doesn't give that many details about what Germain CG tools can do precisely. It allows the user to give his contact information. If it looks serious enough, Yannick Germain sends back a link to a demo.
The direct advertisement is, as far as I know, nonexistent. The important thing is to remain active on the Oracle's support forums and let everyone know that their Siebel related problems can be fixed with Germain CG's tools.
Siebel System employees, even if the company had a ton of employees (about 5,000 in 2005), still have a lot of connections together. People know each other and if a few of them realize that Germain CG's tools can solve many problems, this information can reach most of them.
During 4 days, Germain CG had a booth in the Moscone Center West, where the Oracle Open World '09 took place. Each year, around 60,000 visitors attend this huge conference. Every day of the conference, keynotes were held by CEOs and CIOs of the most important database related companies.
Germain CG makes a lot of business contacts and this year, about 50 prospects came to visit Germain CG's booth.
Ø Hiring a development team
Yannick Germain's choice to hire interns is, to me, a very good choice. As a matter of fact, even their work efficiency is definitely not as high as experienced workers, they can still get the job done. They're also much cheaper and probably the most important thing: they are still learning something. That means that as a project chief, you can give them "your good habits". Whether your habits are really good or not doesn't matter: the important thing is that the interns will become used to your ways of developing.
But of course, at a certain point, interns are not efficient enough. This is why it's also very important to hire experts. Laurent Buhler is an expert developer in Java and Steven Lowenthal is another Siebel expert.
The team is overall quite balanced, mixing experts and interns.
c) Management challenges / Work organization
- Knowledge transfer
- Daily status
- Daily morning meeting
- Monthly 1on1
- No specifications
Yannick Germain is the one that knows the most by far about Siebel and its features. But because of his role, he doesn't have the time to teach everything. So he gives ideas about what needs to be done and puts the development team back on the right track if needed.
Laurent Buhler has another role. He is the one that knows precisely how our software works, since he has built most of its features by himself. On a daily basis, he is able to help the interns on coding issues or the general operation mode of a feature.
Yannick Germain decided, a few months after the beginning of the internship, to set up a support suite. It's a website that allows us to track issues, bugs and enhancements. It also has a customer interface that lets our clients add features they would like to get in the next version. They can, of course, also post bug reports. This greatly helps the development team make sure the dashboard scales correctly with a large amount of data.
Every day, each developer in the development team sends a status of his day. This includes what has been completed and corrected, but also what the issues were, and what the plan is for the next day.
It's directly related to the fact that Yannick Germain is not at the office every day. He's often busy with clients in meetings or simply promoting the software. For that reason, he needs to know as precisely as possible what is going on at the office, on a daily basis.
He is also the development team's best source of knowledge, thanks to his huge experience at Siebel Inc. This is the main reason why it often helps a lot to let him know where we are stuck. The "issues" part of this daily status is the one that has the higher response rate, since he is often the only one able to help.
During the first few months, the interns were attending a daily but short morning meeting with Laurent. He helped us focus on what was the most important thing to achieve on that day. Of course, after a while, this meeting became useless. We were then able to know the most critical issues to solve without asking.
On a monthly basis, each developer in the team would have a "1 on 1" meeting with Yannick Germain. Things like general attitude, work efficiency and general overview of the company and its goals were discussed. These meetings helped me greatly, making sure I was on the right track. They would also make me feel much more part of the company because the near future of the company was often at stake.
I believe that this part is really important, especially for interns. Making sure they feel integrated in the company can only make them more efficient. It's also a good feeling to know that your work is appreciated and that also helped improve my efficiency.
This part is also very "startup specific". Since the company is so small and since Yannick Germain had very often to deal with customers and general promotion of the software, the development team had to make choices without him. The specifications were almost never given, simply to save time. That means that the developers have to find the layout ideas, draw the images and work for the feature from scratch.
This is also really important in the general feeling one can get about his work. It forces the developer to own his features from A to Z, making him feel more responsible for his work.
III. The context
1. Siebel presentation
a) What it is
Siebel CRM Systems, Inc.was asoftwarecompany principally engaged in the design, development, marketing, and support ofcustomer relationship management(CRM) applications.
Customer relationship management(CRM) consists of the processes a company uses to track and organize its contacts with its current and prospectivecustomers. CRM software is used to support these processes; information about customers and customer interactions can be entered, stored and accessed by employees in different company departments. Typical CRM goals are to improve services provided to customers, and to use customer contact information for targeted marketing.
a) What it does – how people use it
Basically, from the user's point of view, it's a webpage that lets you access a lot of information about your company, from the full list of orders to the employees list, and much more!
b) How it works – components, servers...
Siebel, when deployed for a big company, scales with the company's needs. On our development environment, we are only using one server because we are mostly simulating Siebel behavior with a few users. Our only server only runs one component, the call center component. Basically, a Siebel component is a module of the whole application. The screenshot above represents a Siebel component's home page.
But on a production environment, like at our client Netapp, there are about 8000 active users worldwide. That is of course impossible to handle with one server. This is why they have around 15 servers. Each of them is running about 150 components. Call center, data administration, load balancer, dispatcher etc. are components loaded by each server.
2. General thought about Siebel Monitoring / What should be monitored on Siebel CRM?
Before I talk about what needs to be monitored, let me define what I mean by monitoring. Monitoring, as defined by the Webster Dictionary, is to watch, to keep track of, or check usually for a specific purpose. In technical sense, it is the set of activities to gather telemetry data from a piece of hardware or software, analyze the data, and provide some sort of notification if some sort of exceptions are found. Monitoring is closely related to diagnostic. In fact, the same piece of telemetry can be used for both purposes. One might want to monitor CPU usage using data gathered in real time, and examine a time series of CPU trend in diagnosing performance data. Personally, I tend to classify monitoring as the set of tasks that lead to the realization of an exception, and diagnostic as the set of tasks that follow to determine problem root causes.
The obvious things to monitor are CPU, memory, disk space, and I/O (disk, network, etc.). These are the most basic computing resources that Siebel and its underlying database depend on, and they are finite resources, so it makes sense to monitor them. However, these are not the only things, nor are they necessary the most important things.
One thing that makes monitoring Siebel different from monitoring other technologies is that Siebel is an application. As an application, it interacts with users directly, whereas most users do not deal directly with the database, or the load balancer, or the storage devices, and so on. Consequently, the primary purpose of application monitoring is to make sure that the application is providing the service level that users expect in order to do their jobs.
Many things can impact application service level. In fact, every component in a Siebel environment, including but not limited to the Siebel application server, web server, gateway server, report server, CTI, database, storage device, server, network switch, router, load balancer, WAN, etc. can all impact service level. Therefore, it is important to monitor everything, right? Yes and no.
Traditionally, application monitoring means monitoring all the components, and the health of the application is the aggregate health of all the components. However, this kind of bottom up approach is increasingly ineffective because of the increasing amount of redundancy built into production application environments, and because many applications are becoming more and more service oriented. For example, with RAID, it is no big deal to lose a disk. With Oracle RAC, you can lose a database server node and the database will keep on running. With Siebel app server clustering, you can lose an app server altogether but the application would continue to function (yes, users logged onto that server would need to log on again). The point that I want to make is that while it is bad to have component failures, they are not the big catastrophes that they used to be in their service level impacts.
The starting point of Siebel monitoring should be from the top – monitor from the end user perspective by focusing on interactive user sessions and batch jobs, and then move downward to the components. If users have problems accessing application functionalities and getting good response times, or if batch jobs are not getting run within targeted batch window, you clearly have a problem with the application, and those problems may be caused by component level outages. On the other hand, if a server goes down but interactive user sessions and batch jobs are working just fine, you have less to worry about. You'll still want to find out and fix this problem, because the service level of your Siebel environment may drop below your target if another server goes down. Still, the server outage is less urgent than it used to be. In traditional component based monitoring approach, a server outage would be a fatal problem that demanded immediate action. In this top-down end user focused approach, a server outage would most likely be a warning unless there is no redundancy for the component.
Both active and passive approaches should be used for monitoring interactive user workload, and critical alerts should be generated if exceptions occur. For batch workload, the key thing to focus on is whether the job finishes on time and whether errors or warnings are generated in processing the entries. Most of the data that you need to watch are in Siebel log files.
The next set of things to monitor is resources. They are important to monitor because resources tend to be finite. If they run out, processing either stops or is delayed. It's important to keep in mind the relative importance of these resources at the component level though – resource outage may not be a critical event in the grand scheme of things. Traditional resources to monitor include CPU, memory, disk space and I/O, but also Siebel-specific artifacts such as task count. In other words, not only server level CPU, but also CPU consumption specific to the Siebel processes should be monitored.
Lastly, monitor for exceptions, which can be errors showing up on log files, or summarized Siebel server and component statistics for number of level 0 and 1 errors, number of component crashes, restarts, or even number of database connection retries. These are important to monitor in the sense that while a single exception may not be a critical problem, a swamp of these errors happening within a relatively small time window is usually a bad sign, and may point to problems that could cause service level target to be missed.
What about the other Siebel server and component statistics? For the most part, the other statistics are useful for diagnostic and performance tuning purpose. They are not very useful for generating alerts. For example, it is not really practical to set an absolute threshold on a metric such as Average Reply Size, which shows the amount of data Siebel returns. On the other hand, it would be useful to capture the information, and see how the value changes before and after a major application change in order to understand performance impacts. Statistics such as this one should be collected and saved into a database so that trend analysis can be performed.
IV. The choice of the particular subject
1. It's the task I've been given
Adapting technological tools to businesses needs is my everyday job. There is the part where I understand how Siebel users need to have an up and running CRM. There is the part where I understand what our client needs and why monitoring and analyzing Siebel's performance can, at the end, help the Siebel end user. And finally, there is the part where I need to develop, imagine and design the features that will improve Germain CG tools to, at the end again, improve the Siebel end user's experience.
So, on these three levels together, I feel like remodeling, improving and adding new features to the tools is deeply bound to my three years at Supinfo and the idea I have of an IT engineer's duty.
2. It reflects the general idea of the company
Germain CG's most important aspect is to show how adapting Siebel can improve our clients businesses. And our clients know it already. Time is money; every second spent at loading a screen is a waste. And this waste gets even worse when the screen is not loading at all!
V. Presentation of the usual methods used to face the problem
1. The competitors
There are no direct competitors that are offering the same product. But there are a lot of companies that are trying or almost trying to offer what we are offering. Here is a list of the companies and a description of the software they have developed that are the closest to what Germain CG is offering.
Quest's Siebel monitoring tool is much more focused on the error management than we are. At many levels, Siebel generates errors that can be used to monitor and describe problems at a higher level. Below are listed the main features of their "Siebel Management" software.
- Comprehensive Performance Management
- Business Service Grouping
- Intelligent Recording
- Proactive Alerting
Proactively manages performance degradations and application issues before the business is impacted.
Logically groups different activities or services into service groups, allowing application experts and novice administrators to manage the Siebel application.
Records key and routine transactions accessed by Siebel end users to measure and improve application service levels by monitoring actual end user experiences.
Proactively manages service levels for Peoplesoft infrastructure and application components with out-of-the-box alarm configurations.
b) Oracle Enhanced Application Management Pack for Siebel
Oracle Application Management Pack for Siebel is a complete and integrated solution for ensuring the performance and availability of Siebel applications.
- Siebel Workflow Process and Workflow Component Groups
- Siebel Application Services
- Event Log Monitoring
- Configuration Management
- Service Level Management
- Transaction Recording
- Integration with Siebel Transaction Diagnostic
Supports discovery and monitoring of Siebel Workflow Processes and Siebel Workflow Component Groups.
Supports discovery and monitoring of Siebel applications deployed to different servers.
Provides proactive alerts on impending problems on the application.
Allows the support team to view a list of event logs associated with user sessions.
Keeps track of configuration changes and allows the support team to compare configurations across different environments.
Monitors and reports on service levels delivered by the application and performance diagnostics of user and server performance.
Supports recording of Siebel transactions and grouping of related steps into separate, logical entities.
This pack adds integrated Siebel Transaction Diagnostic support using data captured through the Siebel Application Response Measurement (SARM) framework. With this tool, your support team can:
- Analyze request processing for individual user requests.
- Analyze CPU and memory consumption of processing requests for different Siebel Server Components.
c) Mercury Business Availability Center for Siebel business applications
Mercury's solution is mostly based on monitoring what the end user experiences. It therefore has a complete URL simulation engine that allows IT teams to know which click took too long in comparison with the service level agreement.
However, its main drawback is that it doesn't really help to know which click broke the SLA. It would be much more useful if a certain click was bound to a transaction. Here is a short list of Mercury's main features:
- Monitors Siebel application performance and availability from the end-user perspective.
- Allows support teams to receive a notification of potential problems before service-level agreements (SLAs) are broken.
- Quickly diagnoses application problems into the specific tier (Siebel application server, gateway, database, etc.).
IT operation and application teams are able to detect how changes to the underlying infrastructure impact the availability of applications. That means that it checks periodically for Siebel configuration changes.
On a side note, Mercury was bought by HP in 2006.
d) Recursive Technology's "Virtual Admin 2"
Virtual Admin 2 focuses a lot on resources. That means it only deals with a very low level of monitoring. It therefore is very precise concerning CPU, memory and I/O measurements. Below is a list of its main features:
- CPU and memory monitoring
- Usage of Siebel components
- Monitors that detect if your Siebel Services are running
- Siebel Gateway Services
- Siebel Server Services
- Database Availability
To sum it up, the usual methods used to face the problem are very often incomplete. When a competitor thought about implementing the user transactions, he "forgot" to monitor URLs to make sure of what actual end users experience every day.
The only competitor that seems to come out of the pack is Oracle, probably because the company owns Siebel and it has access to the Siebel IT team. But as explained in the next section, Oracle's integrated features are probably more a drawback than an advantage.
VI. Explanation of the approaches chosen to resolve the problem
1. General description of the Germain CG engines
Germain CG engines are processes that are running as background tasks. Their main purpose is to fill a database with values found in the Siebel logs, in configuration files or in the database itself. They can also sometimes send emails, for example to warn our client's support team that something needs to be fixed.
Below are a few examples of Germain CG engines and a short description of what they do:
- Germain Log Analyzer
- Germain SARM Analyzer
- Germain Self Monitoring
- Germain URL simulator
- Germain Siebel Configuration analyzer
This is by far the most important engine. It reads Siebel log files and fills the database with a ton of information, like user transactions names, the time it took to complete the transaction, the SQL queries executed by Siebel during the transaction, the number and type of errors found during the transaction etc.
The SARM (Siebel Analyzer Response Measurement) Analyzer is a very useful tool to go a little deeper into the log files analysis. In fact, Siebel generates SARM files in addition to the log files. The most important information contained in these files is the timeline of each transaction. For example, a SARM file could indicate that the transaction "Login" took 3 seconds. 1 second was used by the web server, 1.5 second by the SQL queries and 0.5 second by the load balancer.
This gives very precise and useful information about how and why the whole transaction was slow.
This engine is mainly used to send email alerts whenever some values are over some thresholds. For example, it would send an email if something was delaying Germain Log Analyzer Engine from working properly. If the server on which this engine is running is overloaded, the log files might not be analyzed fast enough and that could cause problems when trying to display the most recent data.
Basically, this engine monitors the other engines and sends emails whenever something is not running smoothly enough.
Germain tools are meant to help a support team's company. For that reason, it makes sense that an engine is actually simulating what a Siebel user would do. Because when something is not working, it's usually the Siebel end user that notices that a page is not available, or that the login takes 10 minutes!
A Siebel end user usually does the following: launch the browser, log in, click buttons or links on the screen and log out. This is exactly what this engine does. It calls a series of URLs and tracks HTML responses.
The configuration analyzer scans the Siebel configuration file every few minutes and makes sure that nothing has changed. If anything has actually changed, it sends an email to the support team, letting them know that the current configuration does not match the baseline and that either one of Germain engines or even Siebel itself might not work properly anymore.
2. The dashboard
a) The dashboard displays all of the information gathered by the engines.
This is the Dashboard's main window. It displays a lot of information, colored to draw the attention to the critical parts. The main feature of this window is to display the "Siebel User Transactions".
A user transaction is a very important notion to understand. Basically, it represents a user click. Every user made action is logged and the log file is analyzed. This analysis leads to this screen.
For example, the "login" transaction represents the click on the button "login" on the Siebel login page. Therefore, each transaction has a date, a duration, is linked to a user but also to the server it was executed on and the component that was running it.
You can see above that the transactions are displayed and filtered by different categories: Region, Responsibility, Application, User and Transaction Type. These areas on the screen are called indicators, because they indicate and reorder transactions using filters.
Region obviously refers to the region from where the transaction has been logged. It's just as obvious for the User, Application and Responsibility indicators.
The Transaction Type indicator goes a step further. It displays transactions ordered by their type. That means that every "login" transaction for every user on every server and any component will be displayed on the same line.
The colors are also very meaningful. Siebel Administrators have the ability to set up Service Level Agreements (SLA's). They usually set up two values: a red and a yellow threshold. For example, if a login transaction lasts longer than 5 seconds, then the login transaction is RED. If it lasts longer than 2 seconds, then it's YELLOW. If it lasts less, the transaction is GREEN. A special case sometimes occurs, when the transaction is started but never finishes – for example if the server crashes in the middle of the transaction -, then the transaction is BLACK. This is the most critical type of transaction.
The indicators are also divided in two parts: the current and the previous period. This is very helpful to notice the improvement of worsening of the overall availability of Siebel in a day to day analysis.
To sum up, you can see on the screen above that the user "roger" has the reddest transactions. Of all the regions, France is the one that has the slowest transactions. You can also notice that the "login" transaction is never meeting the administrators' expectations.
b) Drilldown on Responsibility
The screen above displays what we call a drilldown. When you click on a pie's section, a list's item or on one of the bars, a new pane appears on the right on the screen.
It displays a lot of very useful data specific to the item you clicked on. In the case above, I clicked on the Responsibility "Sales VP – Executive Level". On the lower pane of the drilldown pane, the detailed analysis shows all the transactions made by users that have this responsibility. You can see very quickly that a lot of the red transactions displayed here are of the type "Login". That tells you two interesting things: first that the problem probably comes from the login transaction and then that it looks like this type of user (Sales VP – Executive Level) probably have to login more often than the other types of users. The solution to this particular problem might be to check if the session timeout for Sales VP is not too low. It might force them to be logged out too often, therefore forcing them to login too often. One could also think that the login server is overloaded, making logins very slow. Actually, this supposition also leads to the timeout variable. It would be because of the timeouts that users have to login too often, making the login server too slow and therefore penalizing all the other login transactions.
The top pane displays the timeline for this group of users. You can see that at midnight, the Siebel Configuration has changed, and that for the next following hour, things have been a little hectic, with a peak at 3am. Times are displayed in GMT. It would probably be better to drill down on a region, but you can already presume that the problems have mainly occurred in the US west time zone. The peak occurred at 6 PM in the Pacific Time zone, which correlates well with the time people have started to log off their sessions. I don't have a precise answer for this specific problem, but the support team should be able to find it using this data.
From this screen, you can also generate a report in PDF or Excel format that you can then send to the support team.
When right clicking on a transaction, you are able to go even further in the analysis. The following screen is called the Siebel Debug screen and gives your very precise information about a specific transaction.
On the first panel are displayed most of the basic information related to the transaction you are debugging. You can see that it is related to the 42 seconds long "Login" transaction. Right below you can see a list of user scenarios. This tells you what kind of other clicks the user has made.
Then, you can even see directly the log file that contains your selected transaction. In this log file, you are able, by using the dropdown buttons, to scroll directly to the interesting areas. As you can see, no errors were found in this transaction. And it looks like the 3 SQL queries made by Siebel for this transaction went fast enough too. The real problem can be identified by using the bottom pane. It shows that of all the Database connections, none of them has used any CPU or memory on the server.
That probably means that there was no data in the user table. The basic automatic behavior in this case for Siebel is to try to find the data somewhere else, on another database center. That explains why this transaction was so slow!
d) Filter Panel
This screen shows a list of all the objects you can filter your data on. As a matter of fact, if your Siebel has a few thousand everyday users and that each of them clicks around 10000 times a day, it become very difficult to display the data you want.
In the example below, the data will be filtered for the day of October 13th for the Region "Finland", the User "antwan" and the Transaction Type "Click on home view (Sales Order)". Simple and efficient.
3. The Engine
a) Log files
Below is an example of a truncated log file that needs to be analyzed. For presentation matters, only a few of them are showed here
In red, the type of Siebel process that is adding this line.
In blue-green, the object the Siebel process is referring to.
In pink, the severity level of this message.
In yellow, the date the message has been written.
In green, the message extension.
In violet, a SQL query.
Transactions can be found by looking for EventContexts in the first word of each line. In the example above, the very last line tells that the user has clicked on the "home page" button.
When Siebel's CRM queries the database, it is shown very clearly by looking at the text right after the sentence SELECT statement in the last group of words of each line.
Siebel errors can be found by looking for the word "Error" in the second group of words of each line. In the example above, on the line before the last line, The user does not exists tells us the kind of error that is generated.
4. è What I've done
Ø URL Simulation
I have developed this feature on my own from scratch. At the start, it was simply a call to some URL that was called every X minutes in a background thread. But I realized that it would not work with our current client's implementation of the login page. As a matter of fact, Netapp is using a secured interface (HTTPS) for their partners' portal. I had to find a way to simulate the complicated cookie exchange between login page, CGI server and the actual partners' home page.
To do that, I first had to be able to sniff the HTML queries that were made, so that I would be able to reproduce them automatically in Java.
This process was very interesting because it forced me to think the entire algorithm through AND to do the research work to analyze how it's working at the same time.
Ø Siebel configuration change monitor
I have developed this feature on my own from scratch. It basically reads a configuration file, transforms it into an xml document. Then it compares the xml document with the one that was created the last time the test ran. And this stack of differences is inserted in a database table, which is at the end displayed in the Dashboard.
The Germain Dashboard user can then decide if he wants those changes to be reverted and only if so, the engine will proceed that order.
The valuable part of this work is, again, the process of finding the right algorithm. It had to be semi automatic because it would otherwise always revert any changes made by our clients.
I have developed this feature on my own from scratch. Using a very modularly way of programming, I have written a daemon that scans the database and looks for abnormal behavior. As previously mentioned, this tool can detect any failures on our engines and send email alerts when needed.
The important part of this project was to make it as modular as possible. With a very few lines of code and with a new line in the database, it's very easy to add a new test. Basically, the engine reads from a table a list of SQL queries, executes them and sends an email if the result is too far from the expected result.
Ø Sarm analyzer
I have built this feature on my own from scratch. It calls a Siebel made executable program that creates a XML file. In this XML file can be found plenty of very useful information about the detailed usage of physical resources.
For example, it would tell you that the "login" transaction made by a certain user on a certain date used up to 10% of CPU, 1Mb of RAM and 50Kb of data were sent over the network. Even more useful, it would tell you that globally, the time spent at dealing with the "login" transaction would be divided like the following:
- 15% CPU
- 20% database query
- 1% load balancer
- 59% data transfer network time
- 5% Web Server HTML generation
The most interesting part of the job was to search online for documentation about this Siebel executable program and to test the XML generation.
Ø Error count
While processing a transaction, the Siebel Server almost always generates errors. Those can have several levels of severity. It's very valuable for the support team to know how many and what type of errors are found during each transaction because when they change the SRF (Siebel Repository File, something that looks like a big configuration file), this number of error might change.
It's for this reason very interesting to know how the support team improved Siebel's performance.
The most valuable part of the development of this feature was the coordination of my work with Loic Gervais' work on the log parsing. We had to work together on a low level part of the Germain Log Engine and that made it even more interesting.
Ø Filtering tools
I completed changed the way data is filtered on the dashboard. The result is explained above in section 2) d) Filter Panel.
I added a new display type for indicators. Previously, they were only available in list or pie display. The bar display is quite different in the data it displays too. Each bar represents an item of the category (for example, a bar represents a User in the "by User" indicator). And the bar is divided in colors. The first part is black, then red, yellow and green. Each segment size corresponds to a percentage of the time spent on black, red, yellow or green transactions for this user.
For example, if the user "Tom" spent 10 seconds on a session and if his login transaction lasted 5 seconds (red transaction) and the two following clicks on the Employees page lasted 2 seconds (yellow transactions), and finally his last click was the logoff button (1 second, green).
It would display a bar that would be half red, two fifths yellow and a tenth green.
The great part of this job was to improve my knowledge of Java displaying possibilities. I learned a lot about how to draw pretty frames.
Ø Smart SQL query builder
I wrote a small program that helps the development team building SQL queries. It turns out to be rather painful to write very long SQL queries and to debug them.
Nothing very interesting, but still useful.
Ø Plenty of UI improvements
I learned a lot about Java Frame designing and Java Layouts, particularly through every developer's best friend: google.com!
VII. Demonstration of the approach's originality
The approach's originality can be explained at several levels:
1. The end user perspective is usually not deep enough in most of the competitor's software.
The first goal of Siebel's monitoring is to improve Siebel end users' efficiency at using Siebel. So, to me, it is particularly important that this end user point of view stands in the few first things to check when checking Siebel's availability.
That means, of course it's important to detect Siebel's failures at the component level. But if the end user doesn't notice it, then it's not as important as a click that takes 5 minutes to react.
Neither Virtual Admin 2, Oracle's Pack for Siebel nor Quest have implemented a feature that actually simulate user clicks on their browsers. Mercury's solution does it, but it has several other drawbacks.
The "login" phase, when the user loads the login page and enters its username and password, can be the source of plenty of errors and delays that will dramatically decrease the user's efficiency if they are not working properly. It's common sense: if the user can't login, then he can't do anything and his productivity is near zero.
Germain CG's URL simulation allows checking that kind of problems, thereby increasing tremendously the end users' efficiency. Of course, same goes for other "user clicks" on Siebel's home page for example.
2. The transaction concept is hardly ever dealt with by most of the competitor's software.
To go a step further, Germain CG's tools are essentially here to help its clients' support team. It makes sense to let them know that this click didn't work, or that the login server is down. But to really help the support team correct these problems, Germain CG goes deeper in the analysis and provides the support team a Siebel Transaction analysis (see above for a complete definition).
This transaction concept is hardly ever mentioned by most of Germain CG's competitors. Only Oracle's solution goes that far.
Virtual Admin 2 only lists resources; Quest is more based on the exceptions that are thrown during these transactions (but without displaying them as a layer in the monitoring interface) and Mercury focuses mostly on the end-user experience.
3. Oracle's solution is the only one that goes as deep as Germain CG tools, but lacks a lot of interoperability.
As a matter of fact, Oracle's Enhanced Application Management Pack for Siebel is only running if Siebel's database is Oracle 10g R2.
This is a shame because only about 40% of the companies that are using Siebel use an Oracle database. And only 80% of them use Oracle 10g R2.
Germain CG tools run just as well if the Siebel database is a DB2, SQL server, MySQL or any JDBC compliant database.
VIII. Reflecting on the management of the thesis
I have encountered quite a lot of problems while writing this thesis. In this section, I will try to be critical about myself and find out how I would have been more efficient.
1. Time management
I have completely underestimated the time it would take to write the thesis. My original plan was to be able to finish it two weeks before the deadline, to give myself some time to think about the parts I wanted to highlight the most.
What really happened is that I was so busy at work that I wasn't able to actually start working on the thesis a month ago. I have pushed back the beginning of the writing only two weeks before the deadline.
The result is not satisfying enough for me. I can feel that my thesis lacks a lot of explanations, especially when trying to demonstrate how I have achieved an engineer's work during my internship.
I could most certainly have made a more describing work about my engineering skills if I wasn't so late at starting to write.
2. Resources management
I guess it's bound to happen to any student working on some very important paper: my laptop died two days before the thesis' deadline. Of course it's only material and luckily, I only lost a couple pages in the process.
There is unfortunately nothing I could have done to avoid that.
3. Documentation management
I haven't taken enough time to gather all I needed for the annexes. I should probably have asked for much more around me to make sure the annexes are complete enough. In fact, I should probably have added some information about the Java public libraries I have used. I should also have detailed the Siebel documentation I have used during the internship that I have found online.
Monitoring Siebel is a complex task to achieve. By making sure the information we gather is relevant and critical for our clients' businesses, we are able to build an image of the product as the most reliable and useful available on the market.
The inherent issues of Siebel, the business issues as well as the number of technical issues are very interesting to solve as an IT engineer because overall, they're widespread. It takes a lot of work of analysis, design and development to make sure our company grows.
The approach to solve the technical issues is very strongly bound to the business' issues solving. In facts, it's only by giving our clients the best image of ourselves at all times that we are able to gain trust. And that trust is a critical aspect in the software world because it's the only way to make our software known worldwide.
I have the feeling that our company can only reach the next stage by continuing the hard work of promoting its software, constantly improving it and giving the image of the most reliable company available in the field.
For this reason, I see myself continuing as long as possible in this company. As a matter of fact, I am going to continue here at least another year and possibly more if the US government allows me to. In 2 years, I will probably feel like programming less and being more involved in the business field. At that time, I'll probably be at a position where I can have a foot in both areas. In 5 years, I will probably want to stop completely the programming to focus entirely on business and management. I could eventually become an expert consultant or vocational trainer.
* Germain CG website: http://www.germaincg.com
* Oracle's website: http://www.oracle.com
* The competitor's websites:
o Quest: http://www.quest.com/application-monitoring-technology/siebel.aspx
o Recursive technologies: http://www.recursivetechnology.com/va2.php
o Mercury Interactive: http://en.wikipedia.org/wiki/Mercury_Interactive
* Siebel Bookshelf: http://www.oracle.com/technology/documentation/siebel.html
* Oracle Open World: http://www.oracle.com/us/openworld/index.htm
Need an essay? You can buy essay help from us today!