Mobilemind

Finally, I feel like I have enough competence on the matter at hand to comment on Tom's blog!

Here's my point: It's a lot easier for to make the content compliant with these standards than it is an LMS (I've been on both sides of the equation).
 

Agreed, building a compliant LMS is tougher. However, a LMS without content isn't viable. Content without an LMS IS useful. One would expect the LMS to do all it can to be content-friendly. An LMS is an expensive line item and I would hope these vendors could afford the investment.
 

An LMS that isn't compliant drives up content costs when it comes to custom courseware. I understand it's difficult, but it's far more important.

Little things like appending spaces to the values returned from fields can wreak havoc and take forever to track down. As a content developer, it's a waste of my time and incredibly frustrating, enough to have me contemplate quitting on the spot.

Bigger problems like poor design can bring down the system. I had a client who's LMS would leave open database sessions if it recieved multiple LMSFinish calls.

I don't want to badmouth them, but Saba's been a culprit numerous times. But they've been crafty enough to turn a profit on it. I have a single client running Saba, which means we have to cough up $1000/year for their Content Lab, which is useless when testing AICC because they only provide content server space for SCO imports.

My (major) client who uses them just started using SCORM a month or two ago for the first time.

Personally, I blame the standards organizations for not throwing their weight around. ADL's SCORM 1.2 RTE has a bug in it where on launch, lesson_location is set to the page name, not an empty string as specified in the CMI spec. It's never been fixed.
 

Tom,

Certification for both standards merely means a third-party has run the standard test suites for the LMS or courseware. Vendors get the same results by running the test suite themselves and publishing the results – which nearly all do. The only advantage to certification is if you think that the vendor would lie to you, cheat on the tests, alter the results, etc.

AICC
The issue here is that the AICC certification expires every five years and yet the standard has not changed significantly in eight or nine years. Many vendors were certified in the past and even those remaining expire this year. Combined with the fact that AICC is not as widely supported, there isn’t a big need for most vendors to certify – you just point to your old certification and say “Nothing changed in the past seven years.”

SCORM
SCORM suffers from almost the opposite. The 2004 standard has had a lot of changes (three editions in four years) and has have compliance and integration issues. Sequencing issues is top on the list of problems and the ADL, vendors, and clients are still trying to get folks to better define behaviors. As such, many vendors are hoping for a better standard/test suite to comply against. Products certified for previous SCORM 2004 editions aren’t too helpful considering the number of “clarifications” in third edition. Even today folks are holding off expecting (and hoping) that the test suite will be updated to something more useful and less likely to change in the future.
 

@bw022-
A few good points there. But what is your larger point, that AICC is too stable and SCORM has changed too much (or is it not enough, in the case of the test tools?). Therefore both are irrelevant and self-policing is all that is needed?

I have high praise for Oracle, Saba and Questionmark who HAVE gone back done the (recurring) homework, (repeatedly) paid the fees and gotten repeated certifications (AICC & SCORM). This doesn't mean any others are liars or cheaters. To me it indicates some of the priorities and diligence of those who get certified. By the way AICC certifications are for 2 years, not 5. Folks like Questionmark have done this all 4 times already--
http://www.aicc.org/pages/press-release-38.htm

What solution do you propose? I'd suggest that a LMS vendor consortium develops open-source versions of the standards test and funds on-going maintenance of that. It would seem to help with the SCORM problems you called out (clarity, cycle times, etc). So far it hasn't happened though. Come to think of it I don't believe there would BE a competitive LMS industry if it wasn't for AICC and SCORM, so content title publishers and LMS vendors might be careful about biting the hands of the AICC and SCORM specs that fed their industry.

One might check to see how many LMS vendors even show up in the LETSI rolls? See--
http://www.letsi.org How many LMS vendors are charter or founding sponsors? Yet this IS the organization that will control future versions of SCORM. You'd think they'd be contributing hundred of thousands of dollars and significant man hours.

"Nothing changed in the last 7 years" seems to imply the meaninglessness of re-certifying. Is upgrading the content or LMS product also unnecessary or irrelevant then? I doubt content or LMS vendors would advocate that.

Or does that mean the code in the product has had zero changes in the last 7 years too? I actually hope that is NOT so. If a house passed inspection 7 years ago, does it still need to be inspected if building codes are the same? Of course. The HOUSE may have changed. Vendors of other systems (LMS or content) and customers need to recognize that regression happens- stuff that worked in an earlier release may get broken in a new release.

Certification programs benefit (content) vendors and suppliers (and customers)-- it adds rigor to the testing and their own preparation. It also improves the standards themselves. For consumers, it indicates a supplier truly have made best efforts and that a 3rd party is able to do produce such results.

Conspiracy theorists may wonder if a vendor's certification (content or LMS) was done under "real-world" circumstances or if it reflects a shipping/publicly available version. Anyone with notepad or an HTML editor can make what looks like valid test results, but that doesn't indicate there was a rigorous, replicable, reliable process behind the displayed results.

If content or LMS vendors are self-certifying and posting it on their own web sites, it sure doesn't seem prominent, nor as complete as the AICC test reports. For example, see the AICC Certification of Saba here--
http://www.aicc.org/pages/cert/report70.pdf

There is good reason for things like JIS, SAE, DIN and other standards groups for automotive specs and a reason workstation vendors created SPEC (http://www.spec.org/spec/)-- certifications are good for the market; customers and vendors alike.

The specs and the tests can be made better. To cite Vince Lombardi, "Gentlemen, we are going to relentlessly chase perfection, knowing full well we will not catch it, because nothing is perfect. But we are going to relentlessly chase it, because in the process we will catch excellence."

Here's to the chase.
 

Tom,

There was no larger point in my post. My two points were: 1) that certification is nothing more than someone else running the test suite and 2) that there are both the AICC and SCORM has current issues which ‘discourage’ folks from certifying.

Your original post said, “Humm, sorta makes you wonder why companies have problems integrating their content with their LMS.”, and your response to mine included statements like “adds rigor”, “truly have made best efforts”, “done under ‘real-world’ circumstances or if it reflects a shipping/publicly available version”, etc. I disagree with the sentiment that certification means any of these.

1. Many certified LMSs/courseware still have issues.

2. Certification does not ensure that the lasted version of the LMS (or even test suite) has been tested against.

3. The certification tests impose no more ‘rigor’ in terms of testing than the self-test. It is the same test. Time and money may demonstrate willingness, and the third-party imparts greater trust, but certification does not mean that a certified product is any less likely to have issues than one self-tested.

4. Certification is not close to ‘best efforts’. Partnering programs, lists of courseware/LMS tested against, running the test suite when it changes or after new releases, testing facilities for third-party courseware, published compliance statements with test results, etc. would all be much higher on my list.

5. Certification does not prevent the vendor from ‘cheating’. They could still hard-code their LMS/courseware behavior to pass the test suites or configure their systems in non-‘real-world’ ways.

Don’t get me wrong – certification is good. It shows a commitment and seriousness to the testing process. However, IMO, that’s all it does. There is no intrinsic benefit from certifying (or recertifying) over self-testing and there are other ways (often more effective) for demonstrating willingness. I’m not sure which vendor I’d feel safer with: vendor A who has say certified for SCORM 2004 2nd-edition, vendor B who has a published SCORM 2004 3rd-edition test suite output, vendor C who is partnered with and tested against my specific courseware, vendor D who is ISO-9000 complaint, etc.?


If you are asking for solutions… I believe that the AICC/ADL would get more vendors (especially courseware vendors) certifying if they offered some substantive benefits and offered clients more substantive reasons to require certification other than a rather vague sense of time and cost equating to better testing processes.

A long time ago, I tossed out the idea that the certification process should give courseware vendors to option of submitting certified courseware to certified LMS vendors. When courseware is certified, the vendor has the option of submitting for specific LMS testing. AICC/ADL posts the courseware such that certified LMS vendors can (if they wish) download it can test it against their LMS. LMS vendor then submits a report which the courseware vendor get look at. Nothing is official – although the courseware vendor should be able to say “Pre-tested against X” if there are no issues. However, there would then be definite benefits to getting certified – LMS vendors get courseware to test against, courseware vendors may have their courseware tested against LMSs, and clients get better lists of courseware tested against LMSs are better for certified products.

If others in the community could come up with substantive benefits and the standards organizations would implement a few, I suspect then there would be greater reasons to certify and stronger reasons to not selected uncertified vendors.