Bug - Freezing on Local Network

Over the week I have set up a presentation which has run through many times on the desktop and the laptop.

Yesterday I set up a local network in the sanctuary using an Airport Express to provide the network.

The Airport Express was wirelessly connected to an Apple TV box (not the latest version but the one before) and the Apple TV to the projector by a HDMI lead. The MacBook computer was connected to the Apple TV using Airplay.

Yesterday I ran through the service in quick time but all seemed to be ok.

Today after about an hour of running the service (30 mins of which was in the PreService Loop) I noticed that the battery on the computer was decreasing rapidly and that there was a warning notice that the laptop was trying to connect to a server. I dismissed this with the red cross.

After a short while the program on the projector froze although the display on the laptop advanced normally.

I quickly went on and off air with no immediate response I then switched the wireless connection on and off. One or the other of these seemed to have the desired effect and I was able to continue with the service.

However the switching brought back the 'trying to connect to server' message which I again dismissed. A short time later the freezing recurred and I repeated the actions - off air - on air - wireless off - wireless on and reestablished the connection.

Hence I am not sure what caused the hiccup - was it my dismissal of the 'trying to connect to server' message - or was it a more general bug?

Nor am I sure what caused the (temporary) regain of control - was it going off air an then back on - was it turning the wi-fi off and then on?

The service was called 'January 10, 2016 Soup'

And here are my logs

0458.Logging.zip

Any insight as to cause and cure would be appreciated.

Find more posts tagged with

Comments

Peter June

Hi, Mike. The good news is that we have all of the same hardware and can test this exact circumstance and see if we get a repro. But we probably won't be able to do that testing until tomorrow.

In the meantime, I've been looking over your logs. They're long, so digesting them will take awhile, but it looks like almost all of the errors I can see so far are just noting that there's no network connectivity. Since the log stretches so many hours, could you please tell me when (roughly) you would have started the on air session?

As far as dismissing the warning, I do not believe that would have affected the presentation. I'll make sure to test this tomorrow, just to ensure there's no bug with that.

Thanks for the detailed report, as always. I'll get back to you as soon as we have some information for you.

Mike Binks

https://community.logos.com/discussion/comment/797132#Comment_797132

Since the log stretches so many hours, could you please tell me when (roughly) you would have started the on air session?

The relevant 'On Air' session would have started just after 09:30hrs GMT on Sunday 17th October 2016. The first failure would have occurred at about 10:30 hrs.

Thanks for looking into this. I have run complete feature films via Airplay to an apple TV (admittedly on a network that does connect to the internet) so I don't think that either Airplay or the Laptops WiFi are likely candidates. Good luck with your testing -- if you find anything I would be prepared to see if I can run a similar operation next week to test any solution.

Mike Binks

https://community.logos.com/discussion/comment/797132#Comment_797132

Hi Peter,

I wonder if there is any update on what went wrong during my presentation,

I hate to say it but ... at the moment... Sunday looms!

Michael Wright

https://community.logos.com/discussion/comment/797870#Comment_797870

We have been using a setup similar to yours for some time now and have experienced this same issue several times. I only have the issue if I forget to restart the MacBook prior to going on-air. We use a rented space and have to set everything up on Sunday mornings. So my pastor usually gives me his sermon notes early; anywhere from 15 minutes to an hour prior to the service start time. Plenty of time to get everything setup and tested just before the service begins! I go on-air long enough to setup the projector during this period, then reboot the Mac prior to going on-air for the actual service. If I don't, The MacBook will loose connectivity with the Apple TV and require a reboot in the middle of the service. This loss of connectivity seems to occur anytime Proclaim is running; whether it is on-air or not. Could there be a memory leak that only occurs when editing content slides? We have watched online movies using this same system with no issues. It only seems to occur when Proclaim is running.

Scott Alexander

https://community.logos.com/discussion/comment/797870#Comment_797870

Hi Peter,

I wonder if there is any update on what went wrong during my presentation,

I hate to say it but ... at the moment... Sunday looms!

Sorry Mike, Peter has been out sick this week and we dropped the ball on this. We'll get back to you on this today sometime.

Mike Binks

https://community.logos.com/discussion/comment/797939#Comment_797939

Michael

Thanks for chipping in. Although in this case 'misery shared' is not 'misery halved' it does help to know that one is not alone in suffering this breakdown.

I only have the issue if I forget to restart the MacBook prior to going on-air. We use a rented space and have to set everything up on Sunday mornings. So my pastor usually gives me his sermon notes early; anywhere from 15 minutes to an hour prior to the service start time. Plenty of time to get everything setup and tested just before the service begins! I go on-air long enough to setup the projector during this period, then reboot the Mac prior to going on-air for the actual service. If I don't, The MacBook will loose connectivity with the Apple TV and require a reboot in the middle of the service.

Mia culpa, I never think to reboot the MacBook, I just open the lid and go. It used to be one of the joys of iOS before faithlife came along. 😊.

I will try and remember to reboot before Sunday service in case the developers haven't got to the bottom of the problem.

Otherwise running via the Apple TV is so convenient - it means that the presenter can be well away from the projector.
The only downside that I have found is that my projector does not have an 'Audio Out' port so I have to split the sound off the HDMI lead using a splitter and that can sometimes introduce a delay that puts lip sync on videos off. No problem for song words, bible readings etc though.

Mike Binks

https://community.logos.com/discussion/comment/797955#Comment_797955

Sorry Mike, Peter has been out sick this week and we dropped the ball on this. We'll get back to you on this today sometime.

Scott - thanks for picking the ball up.

Glenn Gervais (Faithlife)

Mike-

I've been looking through the logs and have run with your presentation (with disabled network) for most of the day. I do see warnings throughout your log file as we attempt to communicate every slide change to our servers (this is how our remote app knows the slide has changed). These timeout very quickly and as you move through the presentation get cleared out. Clearing the 'trying to connect' message along the bottom of Proclaim, simply hides the message, it has no affect of changing whether or not we try to tell our servers that the slide changed. Going OffAir resets our flag for showing the message. All that being said (written), I don't know what caused the projector to freeze up.

01/17/2016 09:34:08 | ERROR | 51 | ApplicationUtility | Error handling response ... Error: ConnectFailure (Network is unreachable)

Proclaim was idle between 8:58 and 9:34:04 and right after Proclaim was woken back up, the network errors started occurring. The network was down. Proclaim continues to run and change slides right up till almost the end of the log file (about 12:11:43, well past going off Air at 11:00:50)

One thing we've chatted about here is if we are leaking connections, and I've not been able to see a leak from Proclaim, maybe another app is using a connection and not releasing it correctly when it times out from the down network. I'm curious if this happens again for you to get a screenshot of your activity monitor showing the threads and ports (switch to Memory tab and right click to add columns).

Hence I am not sure what caused the hiccup - was it my dismissal of the 'trying to connect to server' message - or was it a more general bug?

Nor am I sure what caused the (temporary) regain of control - was it going off air an then back on - was it turning the wi-fi off and then on?

The dismissal of our message simply hides it in the UI, going off air, then back on resets a lot of our internal states and could of free'd up some resources. Turning wi-fi off and on will cause all apps with an open connection to reset and try to re-connect. I suspect this is what allowed the projector (Apple TV) to reconnect to your local network.

Sorry, not much immediate help, but if you get this again, take a screen shot (CMD+Shift+4 brings up cross hair cursor to select area of screen) of your Activity Monitor and grab new logs.

Glenn

Mike Binks

https://community.logos.com/discussion/comment/798082#Comment_798082

Sorry, not much immediate help, but if you get this again, take a screen shot (CMD+Shift+4 brings up cross hair cursor to select area of screen) of your Activity Monitor and grab new logs.

Glenn

Thanks for this Glenn.

I too tried to run the presentation in real time at home using the 'local network' set up. At home, but surrounded by viable internet connections, DSL and 4G, the service seemed to run through with without hiccup.

Hopefully Michael - (see above) may be able to provide some screen shots and logs from his set up for you to compare.

Glenn Gervais (Faithlife)

https://community.logos.com/discussion/comment/797939#Comment_797939

Michael-

I'd be curious to look at your logs after this occurs. Compare to Mike's and my own testing.

http://support.proclaimonline.com/hc/en-us/articles/203763785-How-do-I-submit-log-files-

I'll run some more tests, we do make a lot of network calls, every transition we try to tell our server that the slide changed. If you could also open Activity Monitor (as described for Mike) I'd like to see what your seeing as well.

Thanks

Glenn

Michael Wright

https://community.logos.com/discussion/comment/798246#Comment_798246

I tried to re-create this condition without success for several hours on Friday, Saturday, and for almost an hour this morning while setting up our audio gear and eventually working up sermon notes. I then followed my usual routine and re-started the MacBook prior to going on-air at around 10:15 for service start at 10:45. Even with the re-start, we lost connection with the Apple TV just after finishing the music and starting the sermon. I re-started the MacBook again (it is the only way I have found to re-attach to the Apple TV) and went back on-air and switched to the sermon title slide again. It only worked for a few minutes before loosing connection again. The distraction after this was too much, so I just powered off the projector and continued cycling through slides on the local monitor and recording sermon audio without actually showing the slides on the big screen.

I use the same WIFI and internet connection for remote controlling our QSC TouchMix 16 sound board as I use for connecting to the Apple TV. The QSC remote control software is usually a good indicator when the network is bad, but I never noticed any troubles this morning.

Due to past issues I don't usually use many motion backgrounds, but I used several today. Including one for the sermon title slide, which was active on screen when connection with the Apple TV was lost. Maybe motion backgrounds are part of the cause, but I had tested with these same motion backgrounds for several hours before going live at 10:15 this morning.

So, I don't know if it is the motion backgrounds, or network issues, or something internal to Proclaim, but it happened twice this morning. I did pull the logs after each failure; just before restarting the Mac. Since we were in the middle of our service I did not take the time to pull a screen shot of the activity monitor at the time of failure. I pulled it after getting home from church.

Today's session was "2016-01-24 Sunday"

31364.Proclaim Logs.zip

0435.Proclaim Logs-2.zip

https://www.dropbox.com/s/a142ocl314ixs8j/Screenshot 2016-01-24 14.43.38.png?dl=0

Mike Binks

https://community.logos.com/discussion/comment/798663#Comment_798663

So, I don't know if it is the motion backgrounds, or network issues, or something internal to Proclaim

I suspect that 'motion backgrounds' may turn out to be significant. My instance was the first time in a long time that I moved away from a still-graduated -standard background to incorporate a motion background on all the hymns.

Unfortunately I am a 'jobbing preacher' and so my actual employment of Proclaim is a bit sporadic.

Scott Alexander

https://community.logos.com/discussion/comment/798742#Comment_798742

I'm not sure what we can do on our end to work around this issue. Unless I'm missing something it sounds like an AirPlay issue. I AirPlay to an Apple TV (both 3rd and 4th generation) at home and I have encountered plenty of stability issues. Happy to participate in any troubleshooting but right now we're at a loss for how to proceed.

Michael Wright

https://community.logos.com/discussion/comment/798833#Comment_798833

Are you telling me that the two logs I provided from yesterday show nothing?

Glenn Gervais (Faithlife)

https://community.logos.com/discussion/comment/798846#Comment_798846

Your logs show that Proclaim did not have any problems or issues.

The log from 7:58 to 10:02 had no errors or timeout warnings

The log from 10:02 to 11:20 had one failed web request (timed out talking to our servers)

These are our Proclaim logs, not Airplay or Apple TV logs. I also suspect the motion backgrounds and am theorizing that Apple TV does not handle switching out of short mini-movies as you transition through the presentation. We do not implement anything for AirPlay support, Mac OSX is providing the AirPlay functionality.

Michael Wright

https://community.logos.com/discussion/comment/798860#Comment_798860

Well I was really hoping the logs would show something that would help. But then I am aware of how unreliable Apple TV has been through the years. It just seems strange that this problem only occurs on Sunday mornings. I ran another test using our presentation from Sunday for several hours again last night with no problems. I left is positioned on the sermon title slide (with the motion background) fully expecting it to loose connection with the Apple TV, but it never did. I was watching memory usage the whole time, but never noticed anything unusual. The memory used by Proclaim fluctuated up and down slightly throughout the testing period, but did not indicate any "out-of-control" memory usage or memory leaks. I did notice that memory used by the kernel process increased slightly throughout the testing period.

Maybe our issue is related to the environment or location. Since we only have access to our rented facility for a few hours on Sundays, most of my testing is done at home using the same equipment. I am taking all motion backgrounds out of next Sunday's presentation to see if it makes a difference next week.

Scott Alexander

https://community.logos.com/discussion/comment/799061#Comment_799061

It just seems strange that this problem only occurs on Sunday mornings

I wonder if it could be related to the number of congregants with WIFI enabled devices automatically trying to connect to your wireless access point?

Peter June

https://community.logos.com/discussion/comment/799117#Comment_799117

Michael, I'll have your presentations running on air for the whole day today to see if we can get a repro. I'm guessing that network conditions could have something to do with this problem; I'll run them through some different network configurations to see if that yields anything.

To both of you -- thanks for your patience. I understand it's frustrating to have an obscure problem disrupting your worship. Thank you for working with us to track it down and get fixed.

Peter June

https://community.logos.com/discussion/comment/799145#Comment_799145

Nothing of interest to report to you so far. The presentations have run on several network conditions throughout the day today and none of the conditions appear to have affected the presentation itself. Testing isn't complete on this; I'll keep you updated as we progress.

Michael Wright

https://community.logos.com/discussion/comment/799223#Comment_799223

We had absolutely no issues with Proclaim or the Apple TV/Airplay yesterday; everything worked nicely. I did remove all motion backgrounds from the presentation, and did my usual reboot just prior to going on-air for the service.

I'm still not certain, but am beginning to agree with Scott that this is indeed a WIFI issue, even though we don't have more than five or six people connecting to it on Sunday Mornings. As an additional precaution against service interruption, yesterday I used an AirPort Express as a WIFI repeater (our internet demarcation point, modem and router are located in a closet surrounded by brick walls) and hard-wired the Apple TV using a CAT 5 cable. I believe the AirPort express device has a greater tolerance of weak signals than does the Apple TV. I might try putting some motion backgrounds back in for next week.

So, all this to say, thanks for helping me troubleshoot this, but I don't think you need to spend any more time on it. Please feel free to use the spare time for integrating the local WIFI capability into the remote controls!

Michael Wright

https://community.logos.com/discussion/comment/800359#Comment_800359

So I thought you were done with this issue, but now I am not sure. This mornings service was a train wreck. I watched closely today and took some notes. I rebooted and restarted Proclaim numerous times this morning, but Proclaim never actually crashed, even though Airplay looses connection with the Apple TV. For this reason, and because I didn't have time to grab them, ! have not attached Proclaim logs.

Tested Proclaim on location with Airplay over Apple TV with Airport express as WIFI repeater. Macbook Pro, Presentation 2016-02-07 Sunday.

Proclaim opens at approximately 350mb memory use then begins to climb as the pre-service loop runs. Videos in the PSL do not seem to make any difference. The memory usage seems to ebb and flow, but continues to climb to a point where the videos in the PSL cease to function.

Memory use had climbed to almost 700mb within a few minutes; perhaps 5 minutes.

850mb after about 8 minutes. real memory 666, private memory 500, shared memory 128, cpu usage bouncing around from 14% to 38%

Climbed to 1.10gb memory usage. Real, private and shared memory counts unchanged.

1.26gb. Real, private and shared memory counts unchanged.

1.38gb. Real, private and shared memory counts unchanged.

Time for service now. Rebooting again to clear this memory use.

Airplay failed several songs in during a song with the only motion background in this set.

Rebooted and tried again with same result, but this time I watched the memory usage. I restarted Proclaim and wen on-air from the next song in the set. Everything worked fine until we reached a video element. Proclaim memory use was only about 285mb at the time of Airplay failure, so memory use is apparently not the Airplay issue. Fails after a few seconds of video.

Viewed the video successfully using Airplay, after the service, by viewing it in the Quicktime player with Proclaim still running in the background. No issues doing it this way.

Impressions: memory use increases steadily during the pre-service loop regardless of whether it is displaying a static slide or video. Any long video will kill Airplay. After the preservice loop, memory use continue to climb, but not as quickly. Videos and motion backgrounds do not expedite this increase, but motion backgrounds get real choppy and eventually quit working after the memory usage gets too high. This extreme memory use does not seem to effect Airplay; it fails on any motion background or video.

Proclaim memory use was only about 285mb at one failure, and over 1gb another time. So it would appear that memory use is an issue, but isn’t the Airplay problem.

Jordan Sjodin

https://community.logos.com/discussion/comment/801836#Comment_801836

That memory usage doesnt sound outside the normal range. When you say the videos cease to play, do you mean on the AirPlay device or in proclaim directly?

when using AirPlay in QuickTime, was it using AirPlay directly from QuickTime or via the extended desktop option in the menu bar?

Michael Wright

https://community.logos.com/discussion/comment/801842#Comment_801842

Jordan,

After the memory usage get higher, the videos in the pre-service loop would not display on the big screen via AirPlay and Apple TV. For instance, the Foot Ball Party announcement slide in the Pro Media collection, would quit showing anything other than the detail lines that I had provided on a black screen; time and place,etc. The colorful graphic with goal post and the moving football would disappear.

When I played the 4 minute video using Quicktime this morning, I just clicked on the video file where it was stored in my drop box. This started the Quicktime program and ran the video. I then moved the window over to the big screen and expanded it to full screen; the big screen is the one which is driven by Airplay and Apple TV.

I started a test of today's set locally on the MacBook at about 4:45 this afternoon and went on-air (just with the single MacBook 13 inch display). I started the Activity monitor and watched the memory use grow; I was wondering how high it could get. I have attached a link to a screenshot of the Activity Monitor after running for about 2 Hours and 45 minutes. Proclaim memory usage is in excess of 8 gig. One would thing that the computer would have to eventually run out of swap space and crash, but so far it is still plugging along. I have also attached Proclaims log file for this test but I wouldn't expect you to see anything, since it never crashed.

p.s. I didn't realize until finishing this post, but I think I had switched the motion Foot Ball Party slide out for the static one.

Thanks again for looking into this.

0333.Proclaim Logs 2016-02-07 7-37 PM.zip

https://www.dropbox.com/s/ae5iho11t5lvg16/Screenshot 2016-02-07 19.29.32.png?dl=0

Jordan Sjodin

https://community.logos.com/discussion/comment/801860#Comment_801860

Michael, I'm looking into the memory thing now. I was able to repro on my 13-inch MacBook using your same presentation. One thing to note, in your screenshot, the 'Real Memory' being used by Proclaim was only 269 MBs. This is the reason your system wasn't running out of memory. All the rest was in 'Compressed Memory' which is a fairly new memory management tactic Apple introduced. 'Real memory' is the actually, physical memory that Proclaim was using.

Michael Wright

https://community.logos.com/discussion/comment/801985#Comment_801985

That's interesting. That real memory number was around 666 meg at one point during the fateful morning service. That would have been during the actual service portion of the presentation; probably during, or shortly after, the song with the motion background.