November 17, 2008 SYSTEM PROBLEM - Users are reporting ‘maximum number of cookies’ error when going into Student or Faculty EagleAdvisor, preventing access to EagleAdvisor. A short-term fix is to click on Tools, then Internet Options. In the General tab, click on the Delete option under Browsing History. Then, click on the Delete Cookies. Close all Internet Explorer sessions, reopen, and you will be able to go into Student or Faculty EagleAdvisior. The problem began after system maintenance on Saturday. The problem was not occurring during the testing phase. I.T. is working to correct the problem as quickly as possible. October 22, 2008, we still are seeing intermittent instability in the connection. Outages: 10/21/08 9:06 pm to 10:46 pm - This outage was a little longer I had decided to do some extended testing myself.10/22/08 11:02 am to 11:06 am - Reset Verizon NID equipment connection restored. Verizon was starting to indicate the problem may be with our equipment and I wanted to stop this before it started. During the testing we determined this was not the case and have eliminated this possibility. I was convinced our equipment was not at fault and basic testing confirmed this before contacting Verizon in the first place. During the extended testing I determined what the problem may be or a symptom at least. At the time of the outages their network is functioning with one-way transmission. Normal network operation is always two-way transmission. When we reset the Verizon NID equipment the two-way transmission is restored. Not to go too deep into the technical details. It’s like having a phone call - you can only hear the other person but cannot talk to them. I have called and discussed my test findings with the Verizon group contacts. Verizon will be doing some intrusive testing early Wednesday morning starting at 12:00 am – 1:30 am in an effort to resolve the issue we are having with our wide area network connection for the Menifee Campus. The connection to the Menifee campus will be down between this time. We will be in touch with Verizon during the testing operation. October 21, 2008, Verizon replaced the line card in their Menifee central office equipment on Tuesday from 12:58 am to 2:15 am in an effort to resolve the issue we are having with our wide area network connection for the Menifee campus. The San Jacinto campus and all other locations were not impacted by this outage. We still are seeing intermittent instability in the connection. Outages: 10/21/08 3:36 am to 6:00 am - Reset Verizon NID equipment connection restored 10/21/08 7:47 am to 7:51 am - Reset Verizon NID equipment connection restored 10/21/08 10:09 am to 10:13 am - Verizon replaced the line card in their San Jacinto central office equipment serving the other end of the Menifee connection. This only impacted the Menifee campus and systems hosted at Menifee – all systems are operating normally at this time. The San Jacinto campus and all other locations were not impacted by this equipment change. We are working with Verizon and continue to monitor the situation and instability. October 20, 2008, Verizon has been experiencing a disruption in the central office that our wide area network connection runs through. We have had intermittent outages. Sat 18th12:23pm to 1:17pm, this was due to a power problem with a UPS and not a Verizon problem. Sat 18th4:33pm to 5:51pm, No power Problem – Reset Verizon NID equipment Connection restored maybe UPS problem. Sat 18th9:02pm to 9:46pm, No power problem – Replace UPS Reset Switch Equipment -Reset Verizon NID equipment connection restored. Sun 19th1:05am to 10:19am, Called Verizon put in trouble ticket (1:15AM) Received call from Verizon (around 5:20AM) Verizon to dispatch technicians. Sun 19thMinor interruptions less 15 Min 10:54am to 11:12am Testing. 11:59am to 12:02am, Equipment replacement. 1:20 am to 1:21am Testing. 3:20pm to 3:21PM CO equipment reset. Mon 20th1:17am to 6:46am Reset Verizon NID equipment connection restored. At this time, all system is functional. We may still have intermittent outages until the problem is resolved. Verizon may ask for a 45 Min outage to replace CO equipment. We are in touch with Verizon and we are communicating the urgency for problem resolution. I will update as information comes in. The intermittent Verizon Outages are only impacting the Menifee campus and systems hosted at Menifee. – all systems are operating normally at this time. The San Jacinto campus and all other locations are not impacted by these outages. Verizon will be replacing some Central Office equipment this evening Tuesday 12:00 am to 1:15 am. to try to resolve the issue we are having with our wide area network connection for the Menifee. The connection to Menifee campus will be down between this time. We will be in touch with Verizon during the equipment replacement. The outage will only impact the Menifee campus and systems hosted at the Menifee. The San Jacinto campus and all other locations are not impacted by this outage. We have had one more outage that lasted 3 Minutes at 2:14pm to 2:17pm - Reset Verizon NID equipment connection restored. We will continue to monitor the situation and instability. I will update as information comes in. September 23, 2008. District E-Mail POP3, SMTP and Webmail service was disrupted from early this morning until about 9:00 AM. The problem server and services we're restarted and POP3, SMTP and Webmail service has been restored. September 16, 2008. District e-mail service will be intermittently down the morning of Tuesday September 16th, 2008 from 7:30 to 8:00 PST while we service the e-mail systems. The service is required due to work on an issue with the POP3 service. Student e-mail will not be affected. No data will be lost. All e-mail will be held and delivered once the service is restored. September 10, 2008. District E-Mail POP3 service was disrupted from early this morning until about 9:00 AM. The problem server and services we're restarted and POP3 service has been restored.
August 30, 31 and Sept 1, 2008. During the weekend there was again a disruption in student email live@edu. We are checking to try to find exact times of the disruption. No email was lost.
August 28, 2008. Student email live@edu was intermittently up/down during the day. Microsoft has not given any cause. This system will be monitored. No email was lost. Email may have been delayed. Aug 14, 2008. WebAdvisor hung during the overnight Backup, and was unavailable between 1am and 8:10am, when services were restored. Cause is under investigation.
August 4-9, 2008. A planned Blackboard disruption to upgrade the system. The Blackboard portal and EagleAdvisor will still be available. Students and faculty will not see their Fall 2008 classes in the portal. No adverse affect to add/drop. The system will be operational as normal August 9.
June 18, 2008. We have had an equipment failure this morning that caused a temporary disruption in access to e-mail, web servers, and file servers. The outage only lasted a few minutes. The piece of network equipment that faulted was fault tolerant and took only a few minutes to recover. We do not expect any other outages or disruptions of service as we repair the equipment. You can go to http://www.msjc.edu/it for information about our upcoming projects and system maintenance outages for Summer and Fall.
June 4, 2008. Blackboard disruption at 1pm caused Bb and EagleAdvisor to be down until 2:10pm. Cause was blown electrical circuitry in a Bb UPS. Perhaps an insect got into the case. Resolved, purchased new UPS.
April 22, 2008. The website at http://www.msjc.edu has been down for two hours due to a virus infection. MSJC is mitigating the vulnerability that is in the web content management system implemented in October 2007.
March 5, 2008: The website at http://www.msjc.edu was hit by a Trojan virus. The site had to be blown away and recovered from a backup. The site was down for 18 hours. During that time, traffic was redirected to http://my.msjc.edu
Friday, February 20-Monday March 3 at 8:30pm. The Switched Ethernet Service Line that connects the Menifee and San Jacinto campus went down. This line is owned and operated by Verizon, and leased by the college to provide connectivity between the campuses. During this time, Menifee students, faculty and staff could not access the internet, online classes were not available to students, and no email or Datatel services were provided to the Menifee campus. MSJC administration is communicating with Verizon management regarding the expectation of a higher quality of service.
February 12, 2008 The Datatel system sent a blank student assignments file to blackboard because of incorrect group permissions being set on the JSPARAMS file. The incorrect group permissions appeared to be set through our file analysis that runs at 12:45am. We have taken corrective action by resetting group permissions on the file system after the File Analysis executes. This was already set to happen, but apparently, a typo prevented the update of group permissions. Two corrective actions were taken: 1) Fixed automated permissions update 2) ITS is going to test for file size before import. Impact to blackboard users was between 7:05am and 8:45am. February 7, 2008. Our authentication server for single-sign had an expired security certificate. This problem caused an outage for users attempting to login to my.msjc.edu. The error that users saw while attempting to login was the following "An error occurred in the authentication module, the system is not currently configured for an LDAP referral". The problem was reported to the HelpDesk at 3:00 pm and was resolved by 3:20 pm.
February 1, 2008. Our internet provider experienced an outage at 4:59AM to 6:15 AM on 2/1/08. Some equipment failed in the Los Angles office. The connection may still have some intermittent connectivity. This would have affected all off site access and internet resources. Blackboard, Webadvisor, all email services Staff, Faculty, and Students, and College websites. All local campus services are not affected and are up running.
February 1-2 2008. POP3 services crashed causing the service to stop. This affected offsite e-mail clients connecting to district e-mail boxes via the POP3 connector. Access to web mail was unaffected. The service was restored around 7:00 pm. The problem has been isolated, resolved, and is being monitored for future occurrences.
February 1, 2008. Access to web mail and POP3 service will be down for physical maintenance from 5:00 am to 6:00 am. The server is being relocated to a new location.
January 30-31, 2008. Student e-mail system experienced an outage which affected the ability for students to send e-mail between 5:00 pm to 8:00 am. The students would have received the following message when attempting to send an e-mail "An error occurred while attempting to send the message: bad socket state". The problem has been isolated, resolved, and is being monitored for future occurrences.
January 24th, 2008. Learned the new version of Ipswitch student email was not compatible with Firefox. Students using Firefox have been unable to log in. Problem was determined to be a 'remember' feature of Firefox. Found we were able to override the feature with parms in the email system. Problem corrected.
January 22, 2008. Student email system is unstable. A planned upgrade to a new version of Ipswitch was pushed forward to be performed the evening of the 22nd. Student email was down overnight on the 22nd.
January 22-24, 2008. Datatel system. Major system problems persist with the Unix OS on the Datatel system server. The service patch that HP says will fix our system is crashing it. We are working on failover server, with capacity running 40%-60%. On January 24th, we brought the system up with a new corrective patch, and the AR invoices file was corrupt due to previous system crashes. We turned student billing off to allow students to add/drop. File was corrected on the overnight, and 274 students were rebilled. Calls and emails went to the students notifying them to pay their bill by 7pm on Jan 28th. January 25th, system is restored. January 14, 15, 16, 2007. Datatel system. The mainframe that services the Datatel system for student registrations, drops, transcripts, financial aid, EagleAdvisor, etc., crashed with a Unix operating system error early Monday morning, Jan 14. The system was ported over to a smaller failover system, with 30% access power on the 14th and 70% on the 15th and 16th. The cause of the failure was an operating system failure caused by a bad service patch from Hewlitt Packard. The corrective action taken will be to enable faster failover time.