I was looking at an issue a few days back where a customer complained that when they are selecting a particular IVR option the call gets connected to Music on Hold and then before it gets transferred to an Agent the caller hears “Sorry we are experiencing system problems……” the famous CCX default message when something goes wrong! I am sure you all have come across this dreaded message during your UC Career.
I made some test calls myself and collected logs from SIP Gateway, CCM and CCX to find out where exactly is the call failing.
I looked at the script which was a bit complex as there were several IVR options with nested scripts. This particular case was related to option#4 where CCX will send the call to another script to check opening hours based on Local office number you dialed. I checked IVR logs and I can see the call has gone fine up until this point and got queued correctly. I could also hear Music on hold. This is the section of script which I was interested in.
In IVR logs I am skipping right at the end where Agent was selected and call was delivered to the agent. This is where I heard RINGING tone just after Music on Hold. It just rang once and then I hear “Sorry we are having system problems…..”
## CCX has selected AGENT_3 with extension +4799999999
114292827: May 8 13:05:10.724 GMT %MIVR-SS_RM-7-UNK:[MIVR_SS_RM_RmMsgProcessor-67-0-RmMsgProcessor] Agent: Agent AGENT_3 .setWrapupData(6231017/2, null)
114292829: May 8 13:05:10.724 GMT %MIVR-ICD_CTI-7-UNK:[EventQueue.DispatchThread-0-17] CRACTIEventHandler: EventHandler:got rsrcStateChangedEvent
114292830: May 8 13:05:10.724 GMT %MIVR-ICD_CTI-7-UNK:[EventQueue.DispatchThread-0-17] CRACTIEventHandler: EventHandler: posting {AGENT_STATE_EVENT: Socket:Socket: null monitoredDeviceDN:+4799998888, agentDN:+4799998888, agentID:AGENT_3, monitorID = 0, stateDuration = 0, agentstate = RESERVED, eventreasoncode = 0, agentID = AGENT_3, agentExtension = +4799999999, agentID_Long = AGENT_3 } to outboundQ
114292850: May 8 13:05:10.734 GMT %MIVR-SS_CM-7-UNK:[MIVR_ENG_TASKS-31-14-TASK:0xf2253f431_Script101.aef] TransferCompletionState: Reseting All the flags of CallContact to false
## Here I observed that CCX is complaining about “\” before DN in CUCM but it is without “\” in CCX under Resources. This is not an issue anymore but in older versions of CCX you must have AGENT DN without “\+”
114292851: May 8 13:05:10.734 GMT %MIVR-SS_TEL-7-UNK:[MIVR_ENG_TASKS-31-14-TASK:0xf2253f431_Script101.aef] Port: Agent AGENT_3 Extension contains ‘\’ at the start, ignoring that character : Original Extension=[\+4799999999] New =[+4799999999]
114292852: May 8 13:05:10.734 GMT %MIVR-SS_TEL-7-UNK:[MIVR_ENG_TASKS-31-14-TASK:0xf2253f431_Script101.aef] InCallObserverImpl: CallID:24607 MediaId:6231017/2 Task:65000436785, transfer(+4799999999, 20000, ACKNOWLEDGED)
114292858: May 8 13:05:10.741 GMT %MIVR-SS_TEL-7-UNK:[(P1-172.1.1.1) EventThread] ConsultCallObserver: OrigCall=CallID:24607 MediaId:6231017/2 Task:65000436785, ConsultEvent= ConsultCallActive
114292859: May 8 13:05:10.741 GMT %MIVR-SS_TEL-7-UNK:[(P1-172.1.1.1) EventThread] ConsultCallObserver: OrigCall=CallID:24607 MediaId:6231017/2 Task:65000436785, ConsultEvent= ConnCreatedEv 04720100164:UCCX_P:1
114292860: May 8 13:05:10.741 GMT %MIVR-SS_TEL-7-UNK:[(P1-172.1.1.1) EventThread] ConsultCallObserver: OrigCall=CallID:24607 MediaId:6231017/2 Task:65000436785, ConsultEvent= ConnConnectedEv 04720100164:UCCX_P:1
114292861: May 8 13:05:10.741 GMT %MIVR-SS_TEL-7-UNK:[(P1-172.1.1.1) EventThread] ConsultCallObserver: OrigCall=CallID:24607 MediaId:6231017/2 Task:65000436785, ConsultEvent= CallCtlConnInitiatedEv 04720100164:UCCX_P:1
114292862: May 8 13:05:10.741 GMT %MIVR-SS_TEL-7-UNK:[(P1-172.1.1.1) EventThread] ConsultCallObserver: OrigCall=CallID:24607 MediaId:6231017/2 Task:65000436785, ConsultEvent= TermConnCreatedEv CX_04720100164
114292863: May 8 13:05:10.741 GMT %MIVR-SS_TEL-7-UNK:[(P1-172.1.1.1) EventThread] ConsultCallObserver: OrigCall=CallID:24607 MediaId:6231017/2 Task:65000436785, ConsultEvent= TermConnActiveEv CX_04720100164
114292864: May 8 13:05:10.741 GMT %MIVR-SS_TEL-7-UNK:[(P1-172.1.1.1) EventThread] ConsultCallObserver: OrigCall=CallID:24607 MediaId:6231017/2 Task:65000436785, ConsultEvent= CallCtlTermConnTalkingEv CX_04720100164
114292865: May 8 13:05:10.741 GMT %MIVR-SS_TEL-7-UNK:[(P1-172.1.1.1) EventThread] ConsultCallObserver: OrigCall=CallID:24607 MediaId:6231017/2 Task:65000436785, ConsultEvent= CallCtlConnDialingEv 04720100164:UCCX_P:1
114292866: May 8 13:05:10.742 GMT %MIVR-SS_TEL-7-UNK:[MIVR_SS_TEL_TPG_EXE-40-529158-CALL_EVENT_LOG:04720100164] RequestImpl: CallID:24607 MediaId:6231017/2 Task:65000436785 Got CallCtlConnDialingEv 04720100164:UCCX_P:1, events on the AddressCallObserver.
114292867: May 8 13:05:10.743 GMT %MIVR-SS_TEL-7-UNK:[(P1-172.1.1.1) EventThread] ConsultCallObserver: OrigCall=CallID:24607 MediaId:6231017/2 Task:65000436785, ConsultEvent= CallCtlConnEstablishedEv 04720100164:UCCX_P:1
114292868: May 8 13:05:10.743 GMT %MIVR-SS_TEL-7-UNK:[MIVR_SS_TEL_TPG_EXE-40-529159-CALL_EVENT_LOG:04720100164] RequestImpl: CallID:24607 MediaId:6231017/2 Task:65000436785 Got CallCtlConnEstablishedEv 04720100164:UCCX_P:1, events on the AddressCallObserver.
##Here as you can see I saw an Error “CTIERR_UNSPECIFIED” and this is when I hear “System Error message”
114292869: May 8 13:05:10.746 GMT %MIVR-SS_TEL-7-UNK:[MIVR_ENG_TASKS-31-14-TASK:0xf2253f431_Script101.aef] InCallObserverImpl: CallID:24607 MediaId:6231017/2 Task:65000436785 consultWithoutMedia gets CiscoJtapiException: 0x0(CTIERR_UNSPECIFIED)::Unspecified error
114292870: May 8 13:05:10.747 GMT %MIVR-SS_TEL-7-UNK:[MIVR_SS_TEL_TPG_EXE-40-529160-CALL_EVENT_LOG:04720100164] RequestImpl: CallID:24607 MediaId:6231017/2 Task:65000436785 Got ConnFailedEv 04720100164:UCCX_P:1, CallCtlConnFailedEv 04720100164:UCCX_P:1, events on the AddressCallObserver.
114292875: May 8 13:05:10.938 GMT %MIVR-SS_TEL-7-UNK:[MIVR_ENG_TASKS-31-14-TASK:0xf2253f431_Script101.aef] ConsultCallObserver: OrigCall=CallID:24607 MediaId:6231017/2 Task:65000436785, ConsultEvent= CallObservationEndedEv
114292876: May 8 13:05:10.938 GMT %MIVR-SS_TEL-7-UNK:[MIVR_ENG_TASKS-31-14-TASK:0xf2253f431_Script101.aef] CallImpl: Call.transferFailed(+4799999999, UNKNOWN) JTAPICallContact[id=24607,type=Cisco JTAPI Call,implId=6231017/2
114292879: May 8 13:05:10.939 GMT %MIVR-SS_RM-7-UNK:[MIVR_SS_RM_RmMsgProcessor-67-0-RmMsgProcessor] RsrcMgrMsgProcessor: Processing msg: SessionNotAnsweredMsg
114292880: May 8 13:05:10.939 GMT %MIVR-SS_CM-7-UNK:[MIVR_SS_RM_RmMsgProcessor-67-0-RmMsgProcessor] ContactMgr: ContactMgr.getRmCmContact(6231017/2) returns 39785449 [6231017/2]
114292881: May 8 13:05:10.939 GMT %MIVR-SS_CM-7-UNK:[MIVR_SS_RM_RmMsgProcessor-67-0-RmMsgProcessor] CTIPort: The Resource to which the transfer failed is AGENT_3 in CTIPort 04720100164 .processSessionNotAnsweredMsg (24607, 39785449 [6231017/2])
114292882: May 8 13:05:10.939 GMT %MIVR-SS_RM-7-UNK:[MIVR_SS_RM_RmMsgProcessor-67-0-RmMsgProcessor] RsrcMgr: RsrcMgr.cancelSession(AGENT_3, 39785449 [6231017/2], SESSION_CANCELLED_CAUSE_OTHER)
114292883: May 8 13:05:10.939 GMT %MIVR-SS_RM-7-UNK:[MIVR_SS_RM_RmMsgProcessor-67-0-RmMsgProcessor] RsrcMgrMsgProcessor: Processing msg: SessionCancelledMsg (Rsrc:6231017/2 Cause:SESSION_CANCELLED_CAUSE_OTHER)
114292884: May 8 13:05:10.939 GMT %MIVR-SS_RM-7-UNK:[MIVR_SS_RM_RmMsgProcessor-67-0-RmMsgProcessor] Agent: Agent AGENT_3.processSessionCancelledMsg(SessionCancelledMsg (Rsrc:6231017/2 Cause:SESSION_CANCELLED_CAUSE_OTHER))
I Went back to CUCM to check DN settings for this AGENT_3 and found AGENT had a CALL FORWARD ALL set on the DN. This is a big NO for any CCX Agent as that will cause issues such as above. I removed the Call Forwarding from Agent Extension and calls started to work perfectly fine!
Please note whenever you come across issues like these, always get as much information as possible and most of the time customers describe the problem in a very generic way which would increase your troubleshooting time. Concise and accurate information is inversely proportional to the time you will spend troubleshooting a case. The more concise problem description is the less time you will spend resolving the problem. Even in this case which I discussed above, customer initially said calls to the main number are failing and they are getting that “System error message”. I thought it is failing for all their IVR options so I made some test calls and selected random Menu options and they all worked fine. I then went back to the customer to find out exactly which option is failing and then they confirmed it is option#4. Always get as much information as possible before start troubleshooting as that will save time.
I hope this post was useful. Please like and Subscribe to this post and share.