It’s unusual when something political blips in the technical radar of my news scanning — the NBN-ZTE Scandal being that last major blip — but then it happened: GSIS eyes ‘legal action’ against IBM read an inquirer.net infotech article last Monday. As the story goes, the GSIS is experiencing a constant series of systems crashes, hindering countless man hours of transactions that lead to heavy backlogs. GSIS blames IBM — manufacturer of both the hardware (the servers) and the database software (DB2, or if spelled out “database 2″) for the persistent failures. IBM Philippines, in turn, absolves itself of the problem, blaming a third party vendor instead, for the crashes.
The story was vague as is, so a follow up article was published later: GSIS explains IT systems crash
Explaining the agency’s problematic use of the DB2 software, GSIS chief legal counsel Estrella Elamparo said the software handles tables of up to 250,000 transactions per day.
The agency later discovered that the DB2 tables could only handle 256 gigabytes of data, which was lower than what GSIS needed.
Elamparo said they inquired about the problem with IBM sometime in 2008. The database software was then upgraded to DB2 version 9.1, which could handle 512 exabytes of data, or one billion gigabytes.
The first major crash of the GSIS IT system happened on March 30, 2009 when the agency failed to add new processes from their day’s transactions.
She explained their database table exceeded 2 terabytes (2,000 gigabytes) worth of transactions. “We expected DB2 to keep running since it was only 2 terabytes.”
The system crashes persisted, requiring GSIS to put their extra man-hours to avoid daily backlogs. Saturdays and Sundays were also allocated to update the database.
The agency ordered Questronix to explain the problem and on May 15, the systems integrator said that IBM is working on a special build to permanently remove the persistent crashes.
“Since then, the system crashes we encountered became a daily occurrence. We had to move some of our work to our backup just to continue operating and it’s already frustrating,” Elamparo said.
Business software firm SAP and Questronix were also quoted to have affirmed IBM’s failure in the full page open letter. They have yet to release an official statement on the matter.
IBM stressed that it is still committed to resolving the issue with the GSIS.
Nosebleed. Let’s try to simplify this:
- GSIS had an old version of DB2, with tables that have a maximum capacity of 256 GB. That’s roughly 731 episodes of House from torrents. Or 320 copies of Angels & Demons. Or 64,000 mp3 copies of Careless Whisper. They bought it from IBM years ago.
- GSIS needed the new spanking version of DB2, with each table having a capacity of 512 exabytes. That’s roughly 1,463,000,000,000 episodes of House! Or 128,000,000,000,000 copies of Careless Whisper. You can play with lightballs for the next 70 years!
- GSIS hires Questronix, to figure out how to manage its 128 gazillion copies of Careless Whisper. Questronix uses an application called SAP to do this. SAP, in turn, uses DB2 to hold its data. IBM claims not to know anything about it.
- Unfortunately, new spanking version of DB2 goes pfft. Instead of 128,000,000,000 Careless Whispers, it goes down to only 2 terabytes, or only 500,000 copies of Careless Whisper! Worse, it can only handle 5,714 episodes of House. GSIS gets mad!
- GSIS asks Questronix and the guys who made SAP what’s wrong. Questronix and SAP point fingers at IBM. IBM facepalms — because it wasn’t directly involved with Questronix’s project.
So there’s the rub — IBM is scratching its head over the matter because, as it claims, not only are they helping GSIS figure out the problem, but they were NOT involved with the project in the first place.
From the article alone it’s difficult to figure out what’s the real score on this situation. There are a multitude of problems that could crop up with any super-large database, and these only raises questions. How many people use the database at the same time? How many database servers (e.g., individual machines) are serving these people? How is the load distributed across these servers? How fast are the servers? How big is their memory? How large are the hard disks being used? How fast is the network within GSIS? How efficient are the SAP applications Questronix used to access the data? Lots of questions — and any combination of which answers may hold the key to what is really causing the crashes in GSIS’s system.
The more interesting point is the precedent this incident sets vis a vis IT technology and government. This becomes especially true when you think about the issue of the poll automation technology that is being pushed for next year’s election. This lawsuit will set a precedent as to how such cases will be handled in the future, and consequently, how IT companies will conduct itself with respect to government agencies and involvement — whether direct or indirect — in its critical projects.
Update SAP has denied that it said IBM is to blame for the problems. They also clarified that they did not have a contractual relationship with GSIS. IBM is also stating that they don’t have any working contracts with GSIS.
The way this looks, it’s as if Questronix used a surplus Toyota engine on a jeepney, sold the jeepney to a guy named Winston, and Winston is now suing Toyota Japan because the engine isn’t working properly. Clearly, someone here isn’t telling the truth, or at least, the whole truth.
Popularity: 2% [?]
They sell you Technologies which you cannot modify, or change the
system for better efficiency. Unless you will buy from them.
Maybe, there was a good “lagay” for purchasing the defective system.
Now, the Seller Company has to take back what they have paid for in
“lagay” in technical problems form.
You either junk the whole system. Switch to a new and better system. Or
The seller company can fix it at their own dictated cost and terms. The computer problem will continue for sure…
Exactly:
GSIS did not go to IBM directly but through a 3rd party.
The 3rd party had to downgrade other specs in order to give way to “lagay”.
Now that the system has maxed out, the GSIS puts pressure on the 3rd party – who in turn will point to IBM. To which IBM will say, if you have the budget, we have the solution.
And the 3rd party, eh kasi… yung GSIS… ang lagay eh.. ;)
IBM plays a bit more complete with these things.
Where is the fault?
Where is the culpability?
What does the contract say?
GRP should setup a single IT infrastructure that government agencies can tap for their computing needs. They can start with a DASD farm.
GRP got what it paid for. That’s what happens when you award your contract to the lowest bidder.
As a US based IT (DB2) Pro, I am embarrassed by GSIS and the contractor for allowing the Project to fail. DB2/SAP are both world-class software which are used by bigger clients than GSIS. The 80 million price-tag is too small for a big project like this. Obviously, this project did not go through a bidding process. Otherwise, GSIS would have realized that the winning bid was too low to deliver what they asked for.
Must be buffer overflow
If Buffer problems. They can just expand memories…the system is
not designed to be expanded…
Buffer size needs to be set by the DB administrator. Must be an idiot relative of a politician. The trap file should have logged everything that happened if it was generated. If not generated then the problem is the hardware.
Its a system problem for the DB Administrator. Its a design problem
for both Software and Hardware. It should had been Debugged before
they sold it.
Its a Quality Design problem…GSIS bought the problem for millions
of Pesos. Now GSIS will be like a Stupid Donkey, that does not
know where to choose between two Bales of Hays.
I doubt they didn’t know what they wanted.
They knew what they wanted but someone decided to cut corners. :)
I didn’t catch during my first reading of the article that GSIS upgraded to DB2 version 9. If they really have version 9 now then the DB admin are really morons. DB2 version 9 can adjust almost everything like buffer and storage management. GSIS probably did no have enough physical storage.
there you go, if DB2 v9 is able to meet the requirements, but the hardware has maxed out – the answer then is to beef up the hardware to match the RDBMS capability.
the issue then is not about the specs, bottom line it’s all about the $$$ – and GSIS don’t wanna pay :D
Amen.
It’s corporate tactic to take you by the nose, and empty your
pocket. It’s like milking a good cow (GSIS is a big cow).
Corporate Greed…they call it good…it works, for lack of better
term.
Actually, Jon, you must have heard that several months ago, late last year Metrobank IBM mainframe also went kaput with the system being down for days.
Used to work for Metrobank.
I suppose when the term IBM mainframe is used nowadays, it now refers to the new generations of business computers of IBM, which had their origins with IBM’s system 36/38.
Even then and this was in the 80′s, IBM already called them mini-computers to distinguish from the mainframes we used to imagine – huge machines occupying rooms or entire floors.
Like anything, sooner or later machines will crash. Thus provisions for either back-ups or redundancy.
Aren’t these systems related to those RS-400 (or something) servers… the “mini” mainframes?
System 38/38 was used in the eighties here in the United States.
It is an obsolete system. It has problems regarding the memories,
because of the system designs.
IBM is no longer the forefront of computer innovation in America.
They invented the computer. But Microsoft and other companies
took over the initiative for innovation .
Sales talk is good. If you buy, use your eyes and your brain. Not your ears.
Didn’t hear about the Metrobank crash, but IBM has a plethora of systems/platforms, ranging from mainframe-type machines to more modern types of servers.
So am not sure if it’s precisely the same problem.
What I do know is that these systems (e.g., SAP + IBM DB2) tend to be slow, bloated pieces of software that tries to solve all kinds of problems but end up sucking at any individual task.
The story isn’t unique to IBM, of course. Same complaints have been launched against Microsoft, Sun Microsystems, and other “big” software vendors.
We are talking of System innovation here. Not production.
Companies that has good innovations are the most profitable.
Those who do mass production are just getting by in profit.
This is the problem of the U.S. Auto Industry. Foreign cars
like the Japanese and German cars have more innovations than
American cars. The Sales are reflected on this subject.
I have worked in the auto industry. So, I know it.
I’m wondering the exact configuration of the set up egs, hardware, OS and tier type of storage. Supremo is right for a critical application like this it requires Tier 1 SAN storage like DSK8000 DASD and a higher end hardware preferably in P6-550 to P6-P595 range.
I am working on big datawarehouse with SAP+DB2 9.5 and its not even touching the paging space at all. Performance tuning on OS (AIX in this case) and DB2 are all documented in Red Books. Of course the company has to thrown in tons of $$$ with P550 to P6-P595 hardwares+AIX+ Tier 1 SAN DSK8K and flashcopy for point in time copy/backups and HACMP for failover capability.
Yes, IBM AS-400 came after the 36/38 systems. But in no time sidetracked by the giant leap using PCs for servers and workstations under the OSI model.
I understand AS-400s are still good as DB servers.
An AS-400 is a squat black box no bigger than a tiny cabinet drawer. Thus, the idea of mainframe never really applied to it.
There are better Workstation Mainframes offered by companies,
other than IBM. They are fast, sleek, trouble free and well
designed. Just look around, if you buy..
AS400 can now be an LPAR on a P series hardware and could sit side by side with AIX and Linux on the same physical hardware.
I’m surprised IBM hasn’t been able to solve this yet.
Sure software fails a lot. Remember a few weeks ago, Google failed? But that’s rare especially at the level this requirement of GSIS is suppose to be.
So, let me play devil’s advocate. Is it really the case that IBM screwed up? Could the problem be more than technical? Has there been any mention if GSIS bought the system to do X amount but got the Y package because that’s what they could afford (or want to pay)? I’m too familiar with government as a customer. They want package X, but would like only to pay the Y package.
My point being, there could be another side of the story.
Then if they can’t get X’s performance out of Y, they sue?
I wonder how this goes.
You got it right Jon. the local contractor made the deal and shit happened so GSIS and everyone else points to IBM.
Worst case jon, this goes to court and IBM loses. IBM gets banned from future government contracts. Lucky for IBM they can get lawyers to either throw this out asap. Hey they could have a different wording on their reseller contract. IBM could also have their lawyers drag it out. The cost of litigation for them is after all, minimal.
In 20 years, when the case reaches the Supreme Court, the technology and the matter would be moot, and the biggest loser would be GSIS.
Let me play devil’s advocate again: almost always when such a huge contract is in play, someone profits. Who did?
It will go the way of “areglo”.
Tell the story – just enough to whet people’s attention to the headlines.
If the story gets close to the culprit, a phone call is made, favors are discussed, then the story dies a natural death.
I’m too familiar with government as a customer. They want package X, but would like only to pay the Y package.
MISMO.
I once evaluated an IBM “end-to-end solution” and it looked like a pathetic patchwork of functions cannibalised from products from their various acquisitions and then cobbled together into a “suite”.
The “solution” was so unwieldly that even the IBM sales consultant was often left scratching his head over some of the questions we asked.
But to be fair to IBM, maybe the GSIS allowing Questronix to install an alien application onto DB2 kinda gave IBM an avenue to deny responsibility (kind of like voiding a warranty by attempting an unauthorised repair).
They sold Garbage to a “willing” Buyer. “Lagay” really goes a long
way…
as far as i know, its SAP on DB2 and that GSIS already fired half their IT team… maybe that’s why system is now malfunctioning
dami kasi kumita….
‘The database software was then upgraded to DB2 version 9.1, which could handle 512 exabytes of data, or one billion gigabytes.’
512 exabytes do not mean anything if they don’t have the physical storage to back it up. 250,000 transactions per day is a small volume. Their old DB2 version can handle that volume.
The point here is: Who reviewed the Contract of Sale for GSIS.
The fellow was Negligent, or had sold GSIS to the river…kawawa
naman ang GSIS.
Most likely it’s a GSIS employee, or some corrupt bureaucrat. GSIS has its own procurement department. Go figure.
An inherent problem to the type of transactions that goes on in the Philippines is implementation without testing!
Isa pa yan.
We hope that the GSIS Problem will not infect the COMELEC Automation.
If COMELEC buys “Garbage Equipment”. Naloko na ang 2010 Election!!!
tasio,
That’s precisely one of my concerns. Bobo sa mga ganito ang gubyerno eh.
Its quite obvious that GSIS is looking for a scapegoat. Blame your IT department. Support and maintenance is an integral component for a smooth running system.
you cant blame the software manufacturer..blame the integrators. its like putting different piece of components that does not work harmoniously
I asked IT expert and fellow ZDNet blogger, Paul Murphy, for his opinion on this confusing mess. Paul suggests GSIS owns substantial responsibility:
DB2 can handle the job – but getting AIX to handle large quantities of non local disk is non-trivial and because this particular transition involves migrating people and skills from OS/390 to AIX I’d bet the farm that the technical mistakes causing the failures were made by deeply resentful mainframers doing jobs they don’t understand and don’t want to do – and those people, of course, work for GSIS, not IBM.
Thus senior IBM people in the sales channel probably bear some responsibility for helping the client get into this mess, but blaming either DB2 or IBM’s technical people for it is likely to be wrong.
I asked IT expert and fellow ZDNet blogger, Paul Murphy, for his opinion on this confusing mess. Paul suggests GSIS owns substantial responsibility:
DB2 can handle the job – but getting AIX to handle large quantities of non local disk is non-trivial and because this particular transition involves migrating people and skills from OS/390 to AIX I’d bet the farm that the technical mistakes causing the failures were made by deeply resentful mainframers doing jobs they don’t understand and don’t want to do – and those people, of course, work for GSIS, not IBM.
Thus senior IBM people in the sales channel probably bear some responsibility for helping the client get into this mess, but blaming either DB2 or IBM’s technical people for it is likely to be wrong..
few things i would like to highlight…
(1) there is no BAD PRODUCT.. only bad implementations! WHO IMPLEMENTED THE SYSTEM? WHO IS MAINTAINING IT?
(2) 250K transactions per day is small – i don’t think it’s a limitation, something is fishy… this is not a technical issue…
(3) If this is a critical application for GSIS, are the software on “SUPPORT”? (as IBM claims, it’s not).. Why is it not on support? NO money or they used/ gave/ put the money to somebody else’s pocket?
(4) There is no such thing as free lunch! —
few things i would like to highlight…
(1) there is no BAD PRODUCT.. only bad implementations! WHO IMPLEMENTED THE SYSTEM? WHO IS MAINTAINING IT?
(2) 250K transactions per day is small – i don’t think it’s a limitation, something is fishy… this is not a technical issue…
(3) If this is a critical application for GSIS, are the software on “SUPPORT”? (as IBM claims, it’s not).. Why is it not on support? NO money or they used/ gave/ put the money to somebody else’s pocket?
(4) There is no such thing as free lunch! — They should have contracted IBM Services if QUESTRONIX can’t do the job.
Blame your IT department. Support and maintenance is an integral component for a smooth running system.
if i understand correctly,the rdbms and hardware was setup a few years ago.
needless to say,24/7 disaster recovery (especially daily backups)is a must especially for such a mission critical application.
dba people must monitor disk space,memory and other hardware/software requirements,by coordinating with their inhouse tech/software guys, and the in the inhouse tech/software guys must coordinate with purchasing,and not leave purchasing/procurement or materials management to their own devices.Sorry I have to cite my experience in this area;
been assisting the IT manager of a port operator for quite sometime.
but we must consider,that the government cannot hire personnel to monitor their systems 24 by seven and if an upgrade is necessary,since it is a government run institution,it will take the process to long, so blame cannot be only on the IT department and procurement department.
If in the private sector, it takes a requisition to be approved for months especially if IT was not consulted in the annual budget.
Then a disaster happens, so the blame game begins.
Just my two cents.
another thing to consider is the IT people must do their homework and not let the vendors dictate their destiny.
Lets state the problem “system crash”, system crashes can be software or hardware related. A software crash (that brought down the system) say DB2 generally generates a core dump (if the dump devices were setup, is it questronix?). Core files can be sent out to support for analysis. Usually support people will set up traces and together with the core files try to read what was in memory during the crash. These crashes can be due to library calls that are invalid, device drivers and even processes running on the OS i.e. if these servers were setup as HACMP or highly available clusters. Point is, it might not even be DB2. If a dump was generated it will point you to the right course of action!
As an IBMer here in the US and a Filipino, this is something that is workable if IBM is indeed doing the support. We do this 24×7. Problem is you got Questronix as the middleman and implementation team (they have they’re own Sysadmins and technical support people), am curious if they are on top of they’re game technically.
Exactly. Still points to a how-show lutong makaw transaction between GSIS and questronix. IBM is collateral damage.
Yes, dump files are generated when an error occurs ;it is like an electronic incident report(or log book) which includes time stamps,etc.
In fairness of the so called “after sales support” nothing will come if nothing is asked.
The inhouse IT people must be there.
one entry level staff must be there for the night shift, for first aid remedies and escalation procedures including waking up the supervisors and calling the 24 hour hot lines.
Yes, this is blown out of proportion by the knee jerk reaction of GSIS.
As it was said, a simple dump file sent to the customer service rep of IBM could have done the trick.(if it is really db2)
=============================
ps
due to grammtical errors, it appeared that i still work for the port operator, i mentioned . i am no longer connected with the company for almost seven years.
heard they fired the sys ads and even managers so might be difficult to get history of the implementation. heard also this SAP is highly customized
the thing with exabytes – isn’t this actually the entire database space limit?
and the other thing, with the 2 terrabytes – isn’t is the size of one single table?
if that was a single table, was that even partitioned?
i don’t think we got clear indications from the GSIS spokesperson. either way, they didn’t choose wise on their technical consultants and probably didn’t plan well enough for crisis situations.
Baka nde pa nag comply GSIS sa Licensing agreement(e.g. number of users) sila mayayari…
The articles that was publish should have reasons why the system crash and should state the specifics. (e.g. Systems LOG was not cap to a certain size which made the hardware crash)
Would you think GSIS could hire right people for these kind of projects? Why BDO, BPI, SMART, PLDT, SM and GLobe doesnt have any issues with IBM Systems?
If I’m Winston G. I’d fire all my IT people involved in this project. They are incompetent in evaluating a solution/implementation much more supporting/maintaining it. I can’t imagine the kind of planning they did. 80M for a mission critical system?? SAP alone could cost you 50% of that. Add the required servers, storage devices, redundant devices for backup and the need for a fail over site, application support agreements, etc.. it should have not been 80M in the first place. They cut corners to save major bucks!
It’s a ticking time bomb!
If I were Winston, I won’t fire my IT people. I would rather fire myself. You know why? Well, simple! Being the top official of a government institution like GSIS, I should know better in some, if not in all, areas of database management or systems development.
If it is true that Winston did fire the IT team of GSIS then I would say that he believed he could do it all alone or he might had fired those IT people because somehow they opposed his IT plans for the institution. had he not fired those IT personnel, he could exonerate himself from this fiasco. (grin!)
I am oppose to a total or 100 percent outsourcing of systems development like that of the GSIS and I believe that an institution with a size like that of the GSIS must train and maintain a consirable number of IT personnel not because I don’t trust the vendors but because not all the time the vendors can supply and answer your needs; and not all the time they could understand your problems. some computer systems has to be a “home-grown.” Relying so much on the vendor may compromise your highly confidential data. and besides, vendors just come and go. once the vendors who supplied the system cease to exist, so does the system they had developed and eventually the customer have no choice but to begin from scratch.
let’s face it and let us not so be naive! techinical advices coming from the vendors always self-serving. it will always redound to their own benefit. it is for the look out of the customer to see to it that he/she receives the kind of products or services he/she deserves. one can’t reason out and tell everybody that “tanga kasi ako eh!” … know what u buy.
maybe it’s time for the GSIS to consider and “open source.” (grin!)
i think winston did fire the IT team and hired this pro HP CIO Helen Macaseat
The ALLEGED DB2 fault is that it cannot support 2TB db tables. Which DBA in her right mind would even THINK of implementing such an abomination? How the morty f**k would you back it up? The way to do it, is to partition it across tablespaces, which in turn are in different LUNS in your storage device. It’s more manageable, stable and dang fast.
You see, I’ve been following this thing and have been asking around.
The consultant is an anti-IBM dudette is now pushing GSIS to buy tons of stuff from another provider. Funny though, as all this person does is to present powerpoint stuff made by the other brand. She presents alternatives and let’s GSIS make the decision on what to implement themselves. Funny … i always thought that a consultant is supposed to give you a SOLID recommendation because of experience better than you or your staff has.
However, since her database of choice is not even supported by SAP, she can’t even recommend it.
I think this is just a plot to oust IBM, replace it with their friends’ products and then make tons of money along the way.
HOY! MAAWA NAMAN KAYO SA MGA MIYEMBRO NG GSIS!
We just purchased an IBM machine, should we cancel it? Any advise….
I admit, no software is bug-free and perfect.
But knowing that Philippine govt is involved. Hands down, I know which side is incompetent. Like in every other sector/arena (sports, agrarian, public works, education, public service) the govt is really really amazingly lousy and incompetent(not to mention corrupt).
Clearly, the philippine govt agency knows nothing about purchasing software solutions.
Sorry for the generalization, but I believe that most IT people from govt agencies know nothing about systems programming and have no clue what real acceptance testing is all about. They could’ve done extensive performance testing (Diminishing or exhausting memory, disk, CPU, bandwith resources and checking if the system can still effectively operate).
Most govt IT managers are only interested in biddings(hmmmm.. why do you think so?)
I even doubt they know how to write test specifications or perform code/configuration review for the work done by their IT personnel!
The management people from govt agencies are also tech-clueless and have no practical qualifications to hold such positions. Take a look at the people heading IT-related positions! They dont even know what they are talking about.
Who knows? This statement might be true: There are only three types of people in the govt. The clueless, the Incompetent and the Corrupt.
The GSIS Officers and employees are all foolish brats. They do nothing except plunder the members money!
What I mean were all mostly at Tandag GSIS Satellite Ofdficers. Foolish Indios!