Photon Voice API Change Log v2.57 (December 13, 2024) WebGL: ADDED: WebAudioAudioOut spatial ref and max distances control via AudioSource min and max distances FMOD: FIXED: FMOD always used device 0 for recording v2.56 (August 19, 2024) WebGL: FIXED: WebGL Emscripten Malloc._malloc and Malloc._free replaced with _malloc and _free as these symbols were removed from Module in Emscripten 3.1.31 CHANGED: deprecated Emscripten Module.dynCall_... replaced with makeDynCall macro v2.55 (February 26, 2024) Core: ADDED: visionOS support (Opus and WebRTC Audio libraries) BREAKING: ILogger.Log*() methods replaced with LogLevel enum and single Log() method accepting logging level BREAKING: added ILogger.Level property, that Voice code can check to avoid unnecessary Log() methods calls ADDED: LogLevel.Trace logging level BREAKING: LoadBalancingTransport is no longer ILogger, user code must provide Voice components with ILogger implementation (like Unity.Logger) ADDED: LoadBalancingTransport (and VoiceCLient created by the transport) uses private LBCLogger relying on LoadBalancingClient.DebugReturn() if not provided with an external ILogger FIXED: frameReadPos advance and lost frames detection resulted in false lost frames reports and null frames input to the decoder FIXED: read to write delay was 1 frame larger than set; now if DelayFrames is 0 in unfragmented mode, just written frame can be read immediately FIXED: config frames also put in the normal frame queue to avoid processing it as a lost frame BREAKING: removed VoiceClient.FramesMiss (it's hard to count and not very useful) and FramesLateUsed BREAKING: added IPreviewManager.Has() that allows a ui builder to skip objects that do not have a preview but would otherwise take space in the layout Android: CHANGED: AndroidVideoEncoder: Dispose() called for buffer objects in data callback to release resources earlier FIXED: without objects disposal, the app crashed with 'JNI ERROR (app bug): global reference table overflow' error after a few minutes of running (probably a regression bug) FMOD: FIXED: FMOD.AudioOutEvent playback (OutPos returns the position not wrapped by the event length) v2.54 (January 08, 2024) Core: FIXED: Set() was called sometimes on closed RemoteVoice.frameQueueReady AutoResetEvent FIXED: LoadBalancingTransport sets LoadBalancingClient.ClientType to Voice FIXED: VoiceClient.onVoiceRemove() referenced null in a log message if the voice was not found CHANGED: LoadBalancingTransport.LoadBalancingPeer.ChannelCount is set to at least 4 instead of length of Codec enum + 1 (the enum may be large while we normally do no need more than 4 channels) iOS FIXED: AudioIn: routing audio out to earpiece (receiver) instead of loudspeaker since iOS 17 (and earlier versions for new iPad Pro) due to wrong initialization order (regression): session category is now set after AudioUnit creation CHANGED: AudioIn: when stopping recording, session category is set to its previous values instead of Ambient PS4/PS5: FIXED: the locking in PlayStationAudioOut.cs FMOD: BREAKING: FMOD.AudioInReader no longer replaces -1 device id with 0 assuming that -1 is Voice API default mic: callers should always pass FMOD-specific device id v2.53 (September 06, 2023) Core: FIXED: LoadBalancingTransport: config frame delivery mode is Reliable, the same as for VoiceInfo event, otherwise config frame can be delivered earlier than VoiceInfo FIXED: RemoteVoice: config frames are processed in a separate queue to guarantee that they are decoded: when put in the common queue, they could be dropped if their neighbors have been delivered faster and already processed CHANGED: VoiceClient.ThreadingEnabled is always false for UNITY_WEBGL, setter is ignored ADDED: IDeviceEnumerator.OnReady callback (asynchronous API, required by WebGL): both sync and async implementations call it when the device list is ready CHANGED: all DeviceEnumeratorBase inheritors call OnReady when the list is updated WebGL: ADDED: WebGL video streaming support: video capture, screen capture, rendering to texture, WebCodecs API based encoding and decoding: WebCodecsCameraRecorderUnityTexture, WebCodecsScreenRecorderUnityTexture and WebCodecsVideoPlayerUnityTexture based on WebCodecsCamera, WebCodecsScreenShare, WebCodecsVideoEncoder, WebCodecsVideoDecoderUnityTexture NOTE: browsers supported by WebGL video: Chrome, Opera, Edge ADDED: VideoSourceSizeMode struct with VideoSourceSizeMode.Mode enum controlling the resize of the video source before passing it to the encoder if supported by the source: Fixed (given sizes), Constrained (set to <= given sizes, aspect ratio is preserved) and Source; boolean VideoSourceSizeMode.Update controlling whether the sizes are updated if the source sizes change after initialization (ignored for Fixed) NOTE: VideoSourceSizeMode is currently used only by WebCodecsCameraRecorderUnityTexture / WebCodecsScreenShare ADDED: Platform WebGL video recorder, player and preview factories, 'UnityTexture' methods are 'native' for WebGL ADDED: video input device enumeration: WebVideoInEnumerator FIXED: WebGL Audio AudioWorkletProcessor canceled processing if the first process() call buffer had no data that might happen during initialization, the remote voice did not play in this case ADDED: audio input device enumeration: WebAudioInEnumerator, returns the result asynchronously ADDED: audio input device selection: WebAudioMicIn() accepts 'deviceId' parameter Android: CHANGED: Android native video: KEY_PREPEND_HEADER_TO_SYNC_FRAMES format flag set for h264 (and h265) codec: now every keyframe has vital codec info, unique config frame processing is no longer required CHANGED: AndroidVideoEncoder decoupled from the camera wrapper (ADDED: AndroidCameraVideoEncoder extending AndroidVideoEncoder) and can be used standalone ADDED: AndroidVideoEncoder.Surface property returning the surface to which the input component should write BREAKING: AndroidVideoEncoderSurfaceView -> AndroidCameraVideoEncoderSurfaceView, AndroidVideoEncoderTexture -> AndroidCameraVideoEncoderTexture v2.52 (May 23, 2023) Core: CHANGED: incoming stream events queue reworked to a ring buffer allowing events order restoring (useful for Unsequenced transmission modes), fragmented frames assembly and event cross-referencing for FEC. NOTE: the higher LocalVoice.DelayFrames value, the more chances to use late frames (though saving these frames does not help much Opus while we use quite high 30% PacketLossPercentage value) CHANGED: frame number is added to every event, it's used to run events processing queue instead of event number NOTE: this is the only way to correctly count dropped frames when fragmentation is enabled, which is important for codecs that recover dropped frames, such as Opus (although Opus never uses fragmentation, event and frame numbers are always the same for it) NOTE: advancing at the frame pace is more natural, it allows to keep given frame delay and catch up with bursts of fragmented frames events in time BREAKING: 'frNumber' parameter added to IVoiceTransport.SendFrame(), it can be ignored by SendFrame() implamentation if fragmentation is not used ADDED: FrameBuffer.FrameNum property Fragmentation: ADDED: LocalVoice.Fragment: if true, large frames are fragmented to multiple events by LocalVoice and assembled by RemoteVoice BREAKING: added IVoiceTransport.GetPayloadFragmentSize(): returns the maximum length of the frame data array that fits into one network packet CHANGED: if a fragment is missing, the resulting buffer segment for this fragment is filled with zeroes instead of discarding the entire frame (1st fragment, if delivered, always produces a frame for the decoder, possibly corrupted) CHANGED: Config and KeyFrame are no longer forced to transmit reliably during regular frame sending (but Config is still sent reliable along with voice info when required) FEC: ADDED: LocalVoice.FEC: if > 0, after LocalVoice.FEC outgoing events, a Forward Error Correction event (with the xor of the previous events) is sent Target Players: ADDED: LocalVoice.TargetPlayers: if not null, sending voice info and streaming only to the clients having their player number specified in the array (if supported by the transport), voice info and voice remove are also sent to remote clients on this property update BREAKING: IVoiceTransport.SendFrame(): 'targetMe' and 'targetPlayers' parameters instead of 'targetPlayerId', added SendFrameParams struct parameter holding additional parameters, reference to LocalVoice instance no longer passed ('targetMe' and SendFrameParams are used instead) BREAKING: IVoiceTransport.SendVoicesInfo() -> SendVoiceInfo(): only 1 voice info sending is supported, 'targetMe' and 'targetPlayers' parameters instead of 'targetPlayerId' BREAKING: IVoiceTransport.SendVoiceRemove(): 'targetMe' and 'targetPlayers' parameters instead of 'targetPlayerId' NOTE: 'targetMe' is required because VoiceClient does not know local player number (we could get it from transport but the number is valid only after join) ADDED: transport state change callbacks w/o 'channelId' parameter: VoiceClient.onJoinAllChannels(), onPlayerJoin(int playerId), onPlayerLeave(int playerId) LoadBalancingTransport: CHANGED: stream events delivery mode in SendFrame() changed to ReliableUnsequenced and UnreliableUnsequenced ADDED: 'Voice C++ API compatibility' mode enabled by an optional parameter in constructor: if true, the transport uses sequenced versions (Reliable and Unreliable) of delivery modes in SendFrame() BREAKING: LoadBalancingTransport uses (previously ignored) LocalVoice channelId as Enet channel instead of assigning Enet channel per media type automatically: user must set channelId parameter during LocalVoice creation if channel separation is required Other: BREAKING: removed unused 'channelId' parameter from VoiceClient.onVoiceRemove() and onFrame() BREAKING: LocalVoice parameters set via properties (InterestGroup, TargetPlayers, DebugEchoMode, Reliable, Encrypt, Fragment, FEC) also can be set in LocalVoice constructors and creation methods (LocalVoiceAudio.Create, VoiceClient.CreateLocalVoice, CreateLocalVoiceAudio, CreateLocalVoiceAudioFromSource, CreateLocalVoiceVideo) via new VoiceCreateOptions struct, it's also used to assign Encoder for convenience BREAKING: removed VoiceClient.GlobalInterestGroup, LoadBalancingTransport.GlobalInterestGroup and obsolete LoadBalancingTransport.GlobalAudioGroup, use LocalVoice.InterestGroup and LoadBalancingTransport.OpChangeGroups() to change groups ADDED: LocalVoice.FramesFragmentedSent, FramesFragmentsSent counters ADDED: VoiceClient.FramesRecovered is the number of frames recovered with FEC CHANGED: VoiceClient.FramesLost is the number of empty frames sent to the decoder ADDED: VoiceClient.FramesMiss is the number of slots between correctly ordered frames (how FramesLost was calculated previously) ADDED: VoiceClient.FramesLate is the number of late (incorrectly ordered) frames ADDED: VoiceClient.FramesLateUsed = FramesMiss - FramesLost, the number of late but still used frames NOTE: VoiceClient.FramesLate and VoiceClient.FramesLateUsed are 0 and VoiceClient.FramesMiss == VoiceClient.FramesLost while Unreliable mode is used, they make sense only in UnreliableUnsequences Audio: ADDED: FramerResampler derived from Framer: resamples input data by given ratio before framing it, optional resampling interpolation is available which theoretically improves upsampling quality (in practice noticeable only on tone signal) CHANGED: AudioOutDelayControl delay steps automatic settings are based on Service() call interval (i) instead of frame size, so the state is not changed until playSamplePos is updated in Service(): target is at least i, upper is at least target + i; the resulting delta between steps is not less than set by user CHANGED: default AudioOutDelayControl.PlayDelayConfig.High is 200, the same as Low, this enables automatic tolerance setting NOTE: smaller tolerance prevents significant delay drift from the target value, automatic adjustments set reasonable steps working good in an ideal network even if 0 delays set by user CHANGED: AudioOutDelayControl protected members holding playback parameters made private, implementations should cache them in OutCreate(), now implementations extend AudioOutDelayControl only to implement Out*() interface, they do not use the base class in any other way CHANGED: AudioOutDelayControl.zeroFrame made private, public IsZeroFrame() added CHANGED: AudioOutDelayControl.IsPlaying play detection interval to 120 (60 ms max packet length + some jitter) CHANGED: AudioSyncBuffer rewritten to AudioOutDelayControl implementation CHANGED: WebAudioAudioOut: AudioOutDelayControl.processInService set to false (a frame processed directly in Push()) because Unity API is not used Video: ADDED: ImageBufferNativeGCHandleBytes creating and maintaining byte[] and GHandle per image plane, it replaces obsolete ImageBufferNativeGCHandleSinglePlan Windows: CHANGED: video: MFTVideoEncode sets MF_MT_MAX_KEYFRAME_SPACING encoder output attribute according to Photon_Video_CreateEncoder() 'keyFrameInt' parameter UWP Video Decoder: FIXED: input FrameBuffers are no longer released in MediaStreamSample.Processed which seems to skip some of the samples passed to MediaStreamSource, leading to leaks: we copy the input byte array instead and rely on GC CHANGED: sampleQueue contains byte arrays instead of FrameBuffers FIXED: if sampleQueue is too large, it's cleared and filled with the current frame if it's a keyframe, otherwise the frame is ignored and the queue remains intact (previously it could grow above the limit) CHANGED: sampleQueue max size is 5 to avoid playback delays FIXED: incorrect call order of buffer Retain() prevented buffers from being reused iOS: FIXED: audio capture on iPhone 14: InputCallback is added and used for capture, RenderCallback still exists in the pipeline but does not do anything FIXED: Video: VTCompressionSession ignored bitrate and possibly other properties in recent iOS versions because null timestamp were passed to VTCompressionSessionEncodeFrame(), now the timestamp retrieved from the captured CMSampleBuffer instance is used Mac: CHANGED: Video: current time is used as the timestamp in VTCompressionSessionEncodeFrame() call which looks more reasonable than the previously used frame counter divided by fps (a CMSampleBuffer instance is not available because a byte buffer from an external capture module is passed to the encoder) Android: CHANGED: audio library: all permissions removed from the manifest, it's up to user to set permissions correctly in the application manifest CHANGED: AndroidAudioInParameters turned into struct (with static Default property for default value) and [System.Serializable] attribute added to it to support Unity components fields serialization CHANGED: all AndroidAudioInParameters fields are true by default WebGL: CHANGED: Platform.CreateAudioInEnumerator() returns an instance of the new DeviceEnumeratorSingleDevice calss with the device named 'Default' instead of AudioInEnumeratorNotSupported for Unity WebGL platform CHANGED: Unity.AudioInEnumerator and UnityMicrophone rely on Unity Microphone API in Editor even if the platform is WebGL